diff --git a/2014/01/19/kaggle-beginner-tips/index.html b/2014/01/19/kaggle-beginner-tips/index.html index 09ff6f5d6..3e60da7f5 100644 --- a/2014/01/19/kaggle-beginner-tips/index.html +++ b/2014/01/19/kaggle-beginner-tips/index.html @@ -2,7 +2,7 @@

Kaggle beginner tips

January 19, 2014

These are few points from an email I sent to members of the Data Science Sydney Meetup. I suppose other Kaggle beginners may find it useful.

My first steps when working on a new competition are:

  • Read all the instructions carefully to understand the problem. One important thing to look at is what measure is being optimised. For example, minimising the mean absolute error (MAE) may require a different approach from minimising the mean square error (MSE).
  • Read messages on the forum. Especially when joining a competition late, you can learn a lot from the problems other people had. And sometimes there’s even code to get you started (though code quality may vary and it’s not worth relying on).
  • Download the data and look at it a bit to understand it better, noting any insights you may have and things you would like to try. Even if you don’t know how to model something, knowing what you want to model is half of the solution. For example, in the DSG Hackathon (predicting air quality), we noticed that even though we had to produce hourly predictions for pollutant levels, the measured levels don’t change every hour (probably due to limitations in the measuring equipment). This led us to try a simple “model” for the first few hours, where we predicted exactly the last measured value, which proved to be one of our most valuable insights. Stupid and uninspiring, but we did finish 6th :-). The main message is: look at the data!
  • Set up a local validation environment. This will allow you to iterate quickly without making submissions, and will increase the accuracy of your model. For those with some programming experience: local validation is your private development environment, the public leaderboard is staging, and the private leaderboard is production.
    What you use for local validation depends on the type of problem. For example, for classic prediction problems you may use one of the classic cross-validation techniques. For forecasting problems, you should try and have a local setup that is as close as possible to the setup of the leaderboard. In the Yandex competition, the leaderboard is based on data from the last three days of search activity. You should use a similar split for the training data (and of course, use exactly the same local setup for all the team members so you can compare results).
  • Get the submission format right. Make sure that you can reproduce the baseline results locally.

Now, the way things often work is:

  • You try many different approaches and ideas. Most of them lead to nothing. Hopefully some lead to something.
  • Create ensembles of the various approaches.
  • Repeat until you run out of time.
  • Win. Hopefully.

Note that in many competitions, the differences between the top results are not statistically significant, so winning may depend on luck. But getting one of the top results also depends to a large degree on your persistence. To avoid disappointment, I think the main goal should be to learn things, so spend time trying to understand how the methods that you’re using work. Libraries like sklearn make it really easy to try a bunch of models without understanding how they work, but you’re better off trying less things and developing the ability to reason about why they work or not work.

An analogy for programmers: while you can use an array, a linked list, a binary tree, and a hash table interchangeably in some situations, understanding when to use each one can make a world of difference in terms of performance. It’s pretty similar for predictive models (though they are often not as well-behaved as data structures).

Finally, it’s worth watching this video by Phil Brierley, who won a bunch of Kaggle competitions. It’s really good, and doesn’t require much understanding of R.

Any comments are welcome!

Subscribe
    -
    Alternatively, subscribe to RSS feed.

    Public comments are closed, but I love hearing from readers. Feel free to +

    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

    Flavio
    2014-12-29 13:09:27

    Hi Yanir!

    I have a question.

    When you say: “For example, minimising the mean absolute error (MAE) may require a different approach from minimising the mean square error (MSE).” can you explain what kind of approach (or methods, or rules of thumb) that your get to minimising MAE or MSE in machine learning?

    Thanks for your time in advance!

    Regards,

    Flavio

    Yanir
    2014-12-29 21:50:13

    Hi Flavio!

    The optimisation approach depends on the data and method you’re using.

    A basic example is when you don’t have any features, only a sample of target values. In that case, if you want to minimise the MAE you should choose the sample median, and if you want to minimise the MSE you should choose the sample mean. Here’s proof why: https://www.dropbox.com/s/b1195thcqebnxyn/mae-vs-rmse.pdf

    For more complex problems, if you’re using a machine learning package you can often specify the type of loss function to minimise (see https://en.wikipedia.org/wiki/Loss_function#Selecting_a_loss_function). But even if your measure isn’t directly optimised (e.g., MAE is harder to minimise than MSE because it’s not differentiable at zero), you can always do cross-validation to find the parameters that optimise it.

    I hope this helps.

    Hamid Khan
    2018-10-25 21:18:00

    Hi Yanir!

    appreciate your work! I need to know should I directly jump into machine learning algorithm, programming etc or to first master math and statistics ? I am new in this field.

    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2014/08/17/datas-hierarchy-of-needs/index.html b/2014/08/17/datas-hierarchy-of-needs/index.html index 05fed9ecd..67bd0ba2e 100644 --- a/2014/08/17/datas-hierarchy-of-needs/index.html +++ b/2014/08/17/datas-hierarchy-of-needs/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2014/08/17/datas-hierarchy-of-needs/datas-hierarchy-of-needs.jpg 794w," src=https://yanirseroussi.com/2014/08/17/datas-hierarchy-of-needs/datas-hierarchy-of-needs.jpg alt="Data's hierarchy of needs" loading=lazy>

    In addition, before starting to build a data pipeline, one needs to ensure that the tracked system works as expected. For example, a buggy website is likely to produce weird metrics, which in turn would make the data processing, reporting and predictions unreliable. I completely agree with Jay’s point about needing to get the basis of the pyramid right before setting out to do “something with data” (which seems to be the desire of every company nowadays).

    The general point is that it’s important to have realistic expectations about what can be obtained by data-driven algorithms and insights. These can only be as good as the underlying data, with the results always depending to a large degree on having a solid infrastructure. Not everything has to be perfect from the start (most things never will be), but some degree of robustness is required to avoid spending too many resources on things that would never work. Trying to apply the latest predictive models without a reliable data infrastructure is like driving a fancy car on broken roads – you’re unlikely to get very far.

    Subscribe
      -
      Alternatively, subscribe to RSS feed.

      Public comments are closed, but I love hearing from readers. Feel free to +

      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/08/24/how-to-almost-win-kaggle-competitions/index.html b/2014/08/24/how-to-almost-win-kaggle-competitions/index.html index 76ea34b07..758d27a57 100644 --- a/2014/08/24/how-to-almost-win-kaggle-competitions/index.html +++ b/2014/08/24/how-to-almost-win-kaggle-competitions/index.html @@ -2,7 +2,7 @@

      How to (almost) win Kaggle competitions

      August 24, 2014

      Last week, I gave a talk at the Data Science Sydney Meetup group about some of the lessons I learned through almost winning five Kaggle competitions. The core of the talk was ten tips, which I think are worth putting in a post (the original slides are here). Some of these tips were covered in my beginner tips post from a few months ago. Similar advice was also recently published on the Kaggle blog – it’s great to see that my tips are in line with the thoughts of other prolific kagglers.

      Tip 1: RTFM

      It’s surprising to see how many people miss out on important details, such as remembering the final date to make the first submission. Before jumping into building models, it’s important to understand the competition timeline, be able to reproduce benchmarks, generate the correct submission format, etc.

      Tip 2: Know your measure

      A key part of doing well in a competition is understanding how the measure works. It’s often easy to obtain significant improvements in your score by using an optimisation approach that is suitable to the measure. A classic example is optimising the mean absolute error (MAE) versus the mean square error (MSE). It’s easy to show that given no other data for a set of numbers, the predictor that minimises the MAE is the median, while the predictor that minimises the MSE is the mean. Indeed, in the EMC Data Science Hackathon we fell back to the median rather than the mean when there wasn’t enough data, and that ended up working pretty well.

      Tip 3: Know your data

      In Kaggle competitions, overspecialisation (without overfitting) is a good thing. This is unlike academic machine learning papers, where researchers often test their proposed method on many different datasets. This is also unlike more applied work, where you may care about data drifting and whether what you predict actually makes sense. Examples include the Hackathon, where the measures of pollutants in the air were repeated for consecutive hours (i.e., they weren’t really measured); the multi-label Greek article competition, where I found connected components of labels (doesn’t generalise well to other datasets); and the Arabic writers competition, where I used histogram kernels to deal with the features that we were given. The general lesson is that custom solutions win, and that’s why the world needs data scientists (at least until we are replaced by robots).

      Tip 4: What before how

      It’s important to know what you want to model before figuring out how to model it. It seems like many beginners tend to worry too much about which tool to use (Python or R? Logistic regression or SVMs?), when they should be worrying about understanding the data and what useful patterns they want to capture. For example, when we worked on the Yandex search personalisation competition, we spent a lot of time looking at the data and thinking what makes sense for users to be doing. In that case it was easy to come up with ideas, because we all use search engines. But the main message is that to be effective, you have to become one with the data.

      Tip 5: Do local validation

      This is a point I covered in my Kaggle beginner tips post. Having a local validation environment allows you to move faster and produce more reliable results than when relying on the leaderboard. The main scenarios when you should skip local validation is when the data is too small (a problem I had in the Arabic writers competition), or when you run out of time (towards the end of the competition).

      Tip 6: Make fewer submissions

      In addition to making you look good, making few submissions reduces the likelihood of overfitting the leaderboard, which is a real problem. If your local validation is set up well and is consistent with the leaderboard (which you need to test by making one or two submissions), there’s really no need to make many submissions. Further, if you’re doing well, making submissions erodes your competitive advantage by showing your competitors what scores are obtainable and motivating them to work harder. Just resist the urge to submit, unless you have a really good reason.

      Tip 7: Do your research

      For any given problem, it’s likely that there are people dedicating their lives to its solution. These people (often academics) have probably published papers, benchmarks and code, which you can learn from. Unlike actually winning, which is not only dependent on you, gaining deeper knowledge and understanding is the only sure reward of a competition. This has worked well for me, as I’ve learned something new and applied it successfully in nearly every competition I’ve worked on.

      Tip 8: Apply the basics rigorously

      While playing with obscure methods can be a lot of fun, it’s often the case that the basics will get you very far. Common algorithms have good implementations in most major languages, so there’s really no reason not to try them. However, note that when you do try any methods, you must do some minimal tuning of the main parameters (e.g., number of trees in a random forest or the regularisation of a linear model). Running a method without minimal tuning is worse than not running it at all, because you may get a false negative – giving up on a method that actually works very well.

      An example of applying the basics rigorously is in the classic paper In defense of one-vs-all classification, where the authors showed that the simple one-vs-all (OVA) approach to multiclass classification is at least as good as approaches that are much more sophisticated. In their words: “What we find is that although a wide array of more sophisticated methods for multiclass classification exist, experimental evidence of the superiority of these methods over a simple OVA scheme is either lacking or improperly controlled or measured”. If such a failure to perform proper experiments can happen to serious machine learning researchers, it can definitely happen to the average kaggler. Don’t let it happen to you.

      Tip 9: The forum is your friend

      It’s very important to subscribe to the forum to receive notifications on issues with the data or the competition. In addition, it’s worth trying to figure out what your competitors are doing. An extreme example is the recent trend of code sharing during the competition (which I don’t really like) – while it’s not a good idea to rely on such code, it’s important to be aware of its existence. Finally, reading the post-competition summaries on the forum is a valuable way of learning from the winners and improving over time.

      Tip 10: Ensemble all the things

      Not to be confused with ensemble methods (which are also very important), the idea here is to combine models that were developed independently. In high-profile competitions, it is often the case that teams merge and gain a significant boost from combining their models. This is worth doing even when competing alone, because almost no competition is won by a single model.

      Subscribe
        -
        Alternatively, subscribe to RSS feed.

        Public comments are closed, but I love hearing from readers. Feel free to +

        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

        Toby
        2014-10-08 14:28:21

        Can you elaborate what you mean in Tip 5 by stating “The main scenarios when you should skip local validation is when the data is too small …”. What I experienced is that with too little observations, the leaderboard becomes very misleading, so my intuition would be to use more local validation for small datasets, not less.

        yanirseroussi
        2014-10-08 21:04:52

        Good point. What I was referring to are scenarios where local validation is unreliable.

        For example, in the Arabic writer identification competition (http://blog.kaggle.com/2012/04/29/on-diffusion-kernels-histograms-and-arabic-writer-identification/), each of the 204 writers had only two training paragraphs (all containing the same text), while the test/leaderboard instances were a third paragraph with different content. I tried many forms of local validation but none of them yielded results that were consistent with the leaderboard, so I ended up relying on the leaderboard score.

        Toby
        2014-10-09 06:35:46

        Ah, thanks, that clarifies what you meant. The (currently still running) Africa Soil Property contest (https://www.kaggle.com/c/afsis-soil-properties) seems a bit similar. I won’t put much more energy into that contest, but I am curious how it will work out in the end, and what things will have worked for the winners (maybe not much except pure luck).

        saatvik
        2015-04-11 10:08:23

        Could you provide some tips on #3(‘Getting to Know your data’) with respect to best practice visualisations to gain insights from data - especially considering the fact that datasets always have a large number of features. Plotting feature vs. label graphs do seem to be helpful, but for a large number of features will be impractical. So how should one go about data analysis via visualisation?

        Yanir Seroussi
        2015-04-13 08:29:37

        It really depends on the dataset. For personal use, I don’t worry too much about pretty visualisations. Often just printing some summary statistics works well.

        Most text classification problems are hard to visualise. If, for example, you use bag of words (or n-grams) as your feature set, you could just print the top words for each label, or the top words that vary between labels. Another thing to look at would be commonalities between misclassified instances – these could be dependent on the content of the texts or their length.

        Examples:

        • In the Greek Media Monitoring competition (http://yanirseroussi.com/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/), I found that ‘Despite being manually annotated, the data isn’t very clean. Issues include identical texts that have different labels, empty articles, and articles with very few words. For example, the training set includes ten “articles” with a single word. Five of these articles have the word 68839, but each of these five was given a different label.’ – this was discovered by just printing some summary statistics and looking at misclassified instances
        • Looking into the raw data behind one of the widely-used sentiment analysis datasets, I found an issue that was overlooked by many other people who used the dataset: http://www.cs.cornell.edu/people/pabo/movie-review-data/ (look for the comment with my name – found four years after the original dataset was published)

        I hope this helps.

        saatvik
        2015-05-22 03:11:29

        Thanks a lot, So to summarize the following could be 3

        1. Using summary statistics such as means/stds/variance on the data, looking out for outliers,etc in the data
        2. Looking at misclassified instances during validation to find some sort of pattern in them
        3. Looking at label-specific raw data I apologize for the long overdue response, and thanks for these tips. This will surely be useful in my next Kaggle competition.

        smirshekari
        2015-05-22 01:44:51

        Reblogged this on Dr. Manhattan's Diary and commented: diff --git a/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/index.html b/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/index.html index 9f1641aea..7a108d329 100644 --- a/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/index.html +++ b/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/index.html @@ -2,7 +2,7 @@

        Building a Bandcamp recommender system (part 1 – motivation)

        August 30, 2014

        I’ve been a Bandcamp user for a few years now. I love the fact that they pay out a significant share of the revenue directly to the artists, unlike other services. In addition, despite the fact that fans may stream all the music for free and even easily rip it, almost $80M were paid out to artists through Bandcamp to date (including almost $3M in the last month) – serving as strong evidence that the traditional music industry’s fight against piracy is a waste of resources and time.

        One thing I’ve been struggling with since starting to use Bandcamp is the discovery of new music. Originally (in 2011), I used the browse-by-tag feature, but it is often too broad to find music that I like. A newer feature is the Discoverinator, which is meant to emulate the experience of browsing through covers at a record store – sadly, I could never find much stuff I liked using that method. Last year, Bandcamp announced Bandcamp for fans, which includes the ability to wishlist items and discover new music by stalking/following other fans. In addition, they released a mobile app, which made the music purchased on Bandcamp much easier to access.

        All these new features definitely increased my engagement and helped me find more stuff to listen to, but I still feel that Bandcamp music discovery could be much better. Specifically, I would love to be served personalised recommendations and be able to browse music that is similar to specific tracks and albums that I like. Rather than waiting for Bandcamp to implement these features, I decided to do it myself. Visit BCRecommender – Bandcamp recommendations based on your fan account to see where this effort stands at the moment.

        While BCRecommender has already helped me discover new music to add to my collection, building it gave me many more ideas on how it can be improved, so it’s definitely a work in progress. I’ll probably tinker with the underlying algorithms as I go, so recommendations may occasionally seem weird (but this always seems to be the case with recommender systems in the real world). In subsequent posts I’ll discuss some of the technical details and where I’d like to take this project.


        It’s probably worth noting that BCRecommender is not associated with or endorsed by Bandcamp, but I doubt they would mind since it was built using publicly-available information, and is full of links to buy the music back on their site.

        Subscribe
          -
          Alternatively, subscribe to RSS feed.

          Public comments are closed, but I love hearing from readers. Feel free to +

          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

          Clément
          2019-02-08 15:43:50

          Hi!

          I just found these articles a few years after their publication… I saw that the BCRecommender seems not active anymore and that the last post is from 2015.

          Any update? I’m interested to have your feedback.

          Thanks,

          Clément

          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | diff --git a/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/index.html b/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/index.html index ec342c531..104a3c474 100644 --- a/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/index.html +++ b/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/index.html @@ -2,7 +2,7 @@

          Building a recommender system on a shoestring budget (or: BCRecommender part 2 – general system layout)

          September 7, 2014

          This is the second part of a series of posts on my BCRecommender – personalised Bandcamp recommendations project. Check out the first part for the general motivation behind this project.

          BCRecommender is a hobby project whose main goal is to help me find music I like on Bandcamp. Its secondary goal is to serve as a testing ground for ideas I have and things I’d like to explore.
          One question I’ve been wondering about is: how much money does one need to spend on infrastructure for a simple web-based product before it reaches meaningful traffic?
          The answer is: not much at all. It can easily be done for less than $1 per month.
          This post discusses my exploration of this question by describing the main components of the BCRecommender system, without getting into the algorithms that drive it (which will be covered in subsequent posts).

          The general flow of BCRecommender is fairly simple: crawl publicly-available data from Bandcamp (fan collections and tracks/albums = tralbums), generate recommendations based on this data (static lists of tralbums indexed by fan for personalised recommendations and by tralbum for similarity), and present the recommendations to users in a way that’s easy to browse and explore (since we’re dealing with music it must be playable, which is easy to achieve by embedding Bandcamp’s iframes).

          First iteration: Django & AWS

          The first iteration of the project was implemented as a Django project. Having never built a Django project from scratch, I figured this would be a good way to learn how it’s done properly. One thing I was keen on learning was using the Django ORM with an SQL database (in the past I’ve worked with Django and MongoDB). This ended up working less smoothly than I expected, perhaps because I’m too used to MongoDB, or because SQL forces you to model your data in unnatural ways, or because I insisted on using SQLite for simplicity. Whatever it was, I quickly started missing MongoDB, despite its flaws.

          I chose AWS for hosting because my personal account was under the free tier, and using a micro instance is more than enough for serving a website with no traffic. I considered Google App Engine with its indefinite free tier, but after reading the docs I realised I don’t want to jump through so many hoops to use their system – Google’s free tier was likely to cost too much in pain and time.

          While an AWS micro instance is enough for serving the recommendations, it’s not enough for generating them. Rather than paying Amazon for another instance, I figured that using spare capacity on my own laptop (quad-core with 16GB of RAM) would be good enough. So the backend worker for BCRecommender ended up being a local virtual machine using one core and 4GB of RAM.

          After some coding I had a nice setup in place:

          • AWS webserver running Django with SQLite as the database layer and a simple frontend, styled with Bootstrap
          • Local backend worker running Celery under Supervisor to collect the data (with errors reported to a dedicated Gmail account), Dropbox for backups, and Django management commands to generate the recommendations
          • Code and issue tracker hosted on Bitbucket (which provides free private repositories)
          • Fabric scripts for deployments to the AWS webserver and the local backend worker (including database sync as one big SQLite file)
          • Local virtual machine for development (provisioned with Vagrant)

          This system wasn’t going to scale, but I didn’t care. I just used it to discover new music, and it worked. I didn’t even bother registering a domain name, so it was all running for free.

          Second iteration: “Django” backend & Parse

          A few months ago, Facebook announced that Parse’s free tier will include 30 requests / second. That’s over 2.5 million requests per day, which is quite a lot – probably enough to run the majority of websites on the internet. It seemed too good to be true, so I had to try it myself.

          It took a few hours to convert the Django webserver/frontend code to Parse. This was fairly straightforward, and it had the added advantages of getting rid of some deployment scripts and having a more solid development environment. Parse supplies a command-line tool for deployment that constantly syncs the code to an app that is identical to the production app – much better than the Fabric script I had.

          The disadvantages of the move to Parse were having to rewrite some of the backend in JavaScript (= less readable than Python), and a more complex data sync command (no longer just copying a big SQLite file). However, I would definitely use it for other projects because of the generous free tier, the availability of APIs for all major platforms, and the elimination of most operational concerns.

          Current iteration: Goodbye Django, hello BCRecommender

          With the Django webserver out of the way, there was little use left for Django in the project. It took a few more hours to get rid of it, replacing the management commands with Commandr, and the SQLite database with MongoDB (wrapped with the excellent MongoEngine, which has matured a lot in recent years). MongoDB has become a more natural choice now, since it is the database used by Parse. I expect this setup of a local Python backend and a Parse frontend to work quite well (and remain virtually free) for the foreseeable future.

          The only fixed cost I now have comes from registering the bcrecommender.com domain and managing it with Route 53. This wasn’t required when I was running it only for myself, and I could have just kept it under bcrecommender.parseapp.com, but I think it would be useful for other Bandcamp users. I would also like to use it as a training lab to improve my (poor) marketing skills – not having a dedicated domain just looks bad.

          In summary, it’s definitely possible to build simple projects and host them for free. It also looks like my approach would scale way beyond the current BCRecommender volume. The next post in this series will cover some of the algorithms and general considerations of building the recommender system.

          Subscribe
            -
            Alternatively, subscribe to RSS feed.

            Public comments are closed, but I love hearing from readers. Feel free to +

            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/index.html b/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/index.html index 1d2a8a2f3..04c1c0899 100644 --- a/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/index.html +++ b/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/psychedelic-progressive-rock-tag-cloud.png 960w," src=https://yanirseroussi.com/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/psychedelic-progressive-rock-tag-cloud_hu1f3068901702be9a79eb77586e667f5e_136685_800x0_resize_box_3.png alt="psychedelic-progressive-rock tag cloud" loading=lazy>

            Using LDA for generating recommendations is straightforward, as each fan can be represented as the concatenation of the tags assigned to their tralbums, together with their own user tag. This representation is then converted to a topic distribution, which is compared to all the tralbums to yield the most similar ones.

            This approach yielded much better results than collaborative filtering. I actually found albums I like and made some purchases, more than just the three that are annotated on my fan page (I didn’t want to be too spammy). Woohoo!

            However, the problem with this approach is that it doesn’t take my mood into account, as it is based on my entire profile. To address this, I introduced similar music and cluster-based discovery.

            Beyond static personalisation: similar music and cluster-based discovery

            It is easy to see that the LDA-based tralbum representation makes it straightforward to calculate similarity between tralbums, and also explore tralbums that belong to the same topic/cluster. Adding this functionality to BCRecommender means that users can explore similar tralbums to a tralbum or a cluster in the style that they are interested in right now – based on their mood. Implementing these features helped me find more music I like, so again, I’m happy.

            Tweaking the similarity algorithms is still a work in progress, as is finding a scalable way to generate useful cluster/spotlight pages. However, my focus now (in the time that I can allocate to working on this project) is on getting some people using it and iterate following their feedback.

            Future extensions

            It would be awesome to make BCRecommender’s discovery process smoother. For example, it’d be fairly straightforward to just stream all the recommendations rather than making users click album by album (like Pandora, Spotify, etc.). Iterating on the above approaches to improve the user experience is also likely to yield good results.

            However, as mentioned above, my current focus is on getting more people to use BCRecommender. While the target audience is rather small, it doesn’t matter because I’m not trying to make money from this project. I am certain that many fans would discover new music using the website. At this stage, I just need to get them to visit, which is something that I will write about in future posts.

            Subscribe
              -
              Alternatively, subscribe to RSS feed.

              Public comments are closed, but I love hearing from readers. Feel free to +

              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/index.html b/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/index.html index 2838c4405..78d367c5c 100644 --- a/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/index.html +++ b/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/index.html @@ -2,7 +2,7 @@

              Applying the Traction Book’s Bullseye framework to BCRecommender

              September 24, 2014

              This is the fourth part of a series of posts on my Bandcamp recommendations (BCRecommender) project. Check out previous posts on the general motivation behind this project, the system's architecture, and the recommendation algorithms.

              Having used BCRecommender to find music I like, I’m certain that other Bandcamp fans would like it too. It could probably be extended to attract a wider audience of music lovers, but for now, just getting feedback from Bandcamp fans would be enough. There are about 200,000 fans that I know of – getting even a fraction of them to use and comment on BCRecommender would serve as a good guide to what’s worth building and improving.

              In addition to getting feedback, the personal value for me in getting BCRecommender users is learning some general lessons on traction building. Like many technical people, I like building products and playing with data, but I don’t really enjoy sales and marketing (and that’s an understatement). One of my goals in working independently is forcing myself to get better at the things I’m not good at. To that end, I recently started reading Traction: A Startup Guide to Getting Customers by Gabriel Weinberg and Justin Mares.

              The Traction book identifies 19 different channels for getting traction, and suggests a simple framework (named Bullseye) to ranking and quickly exploring the channels. They explain that many technical founders tend to focus on traction channels they’re familiar with, and that the effort invested in those channels tends to be rather small compared to the investment in building the product. The authors rightly note that “Almost every failed startup has a product. What failed startups don’t have is traction – real customer growth.” They argue that following a rigorous approach to gaining traction via their framework is likely to improve a startup’s chances of success. From personal experience, this is very likely to be true.

              The key steps in the Bullseye framework are brainstorming ideas for each traction channel, ranking the channels into tiers, prioritising the most promising ones, testing them, and focusing on the channels that work. This is not a one-off process – channel suitability changes over time, and one needs to go through the process repeatedly as the product evolves and traction grows.

              Here are the traction channels, ordered in the same order as in the book. Each traction channel is marked with a letter denoting its ranking tier from A (most appropriate) to C (unsuitable right now). A short explanation is provided for each channel.

              • [B] viral marketing: everyone wants to go viral, but at the moment I don’t have a good-enough understanding of my target audience to seriously pursue this channel.
              • [C] public relations (PR): I don’t think that PR would give me access to the kind of focused user group I need at this phase.
              • [C] unconventional PR: same as conventional PR.
              • [C] search engine marketing (SEM): may work, but I don’t want to spend money at this stage.
              • [C] social and display ads: see SEM.
              • [C] offline ads: see SEM.
              • [A] search engine optimization (SEO): this channel seems promising, as ranking highly for queries such as “bandcamp recommendations” should drive quality traffic that is likely to convert (i.e., play recommendations and sign up for updates). It doesn’t seem like “bandcamp recommendations” is a very competitive query, so it’s definitely worth doing some SEO work.
              • [A] content marketing: I think that there’s definitely potential in this channel, since I have a lot of data that can be explored and presented in interesting ways. The problem is creating content that is compelling enough to attract people. I started playing with this channel via the Spotlights feature, but it’s not good enough yet.
              • [B] email marketing: BCRecommender already has the subscription feature for retention. At this stage, this doesn’t seem like a viable acquisition channel.
              • [B] engineering as marketing: this channel sounds promising, but I don’t have good ideas for it at the moment. This may change soon, as I’m currently reading this chapter.
              • [A] targeting blogs: this approach should work for getting high-quality feedback, and help SEO as well.
              • [C] business development: there may be some promising ideas in this channel, but only worth pursuing later.
              • [C] sales: not much to sell.
              • [C] affiliate programs: I’m not going to pay affiliates as I’m not making any money.
              • [B] existing platforms: in a way, I’m already building on top of the existing Bandcamp platform. One way of utilising it for growth is by getting fans to link to BCRecommender when it leads to sales (as I’ve done on my fan page), but that would be more feasible at a later stage with more active users.
              • [C] trade shows: I find it hard to think of trade shows where there are many Bandcamp fans.
              • [C] offline events: probably easier than trade shows (think concerts/indie events), but doesn’t seem worth pursuing at this stage.
              • [C] speaking engagements: similar to offline events. I do speaking engagements, and I’m actually going to mention BCRecommender as a case study at my workshop this week, but the intersection between Bandcamp fans and people interested in data science seems rather small.
              • [C] community building: this may be possible later on, when there is a core group of loyal users. However, some aspects of community building are provided by Bandcamp and I don’t want to compete with them.

              Cool, writing everything up explicitly was actually helpful! The next step is to test the three channels that ranked the highest: SEO, content marketing and targeting blogs. I will report the results in future posts.

              Subscribe
                -
                Alternatively, subscribe to RSS feed.

                Public comments are closed, but I love hearing from readers. Feel free to +

                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/index.html b/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/index.html index a85ca68b1..ba1cdf728 100644 --- a/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/index.html +++ b/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/wise2014-connected-components.png 769w," src=https://yanirseroussi.com/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/wise2014-connected-components.png alt="wise2014 connected components" loading=lazy>

                My best submission ended up being a simple weighted linear combination of three models. All these models are hierarchical ensembles, where a linear classifier distinguishes between connected components, and the base models are trained on texts from a single connected component. These base models are:

                1. Ensemble of classifier chains (ECC) with linear classifiers (SGDClassifier from scikit-learn) trained for each label, using hinge loss and L1 penalty
                2. Same as 1, but with modified Huber loss
                3. A linear classifier with modified Huber loss and L1 penalty that predicts single label probabilities

                For each test document, each one of these base models yields a score for each label. These scores are weighted and thresholded to yield the final predictions.

                It was interesting to learn that a relatively-simple model like ECC yields competitive results. The basic idea behind ECC is to combine different classifier chains. Each classifier chain is also an ensemble where each base classifier is trained to predict a single label. The input for each classifier in the chain depends on the output of preceding classifiers, so it encodes dependencies between labels. For example, if label 2 always appears with label 1 and the label 1 classifier precedes the label 2 classifier in the chain, the label 2 classifier is able to use this dependency information directly, which should increase its accuracy (though it is affected by misclassifications by the label 1 classifier). See Read et al.’s paper for a more in-depth explanation.

                Another notable observation is that L1 penalty worked well, which is not too surprising when considering the fact that the dataset has 300K features and many of them are probably irrelevant to prediction (L1 penalty yields sparse models where many features get zero weight).

                Things that didn’t work

                As I was travelling, I didn’t have much time to work on this competition over its two final weeks (though this was a good way of passing the time on long flights). One thing that I tried was understanding some of the probabilistic classifier chain (PCC) code out there by porting it to Python, but the results were very disappointing, probably due to bugs in my code. I expected PCC to work well, especially with the extension for optimising the F-measure. Figuring out how to run the Java code would have probably been a better use of my time than porting the code to Python.

                I also played with reverse-engineering the features back to counts, but it was problematic since the feature values are normalised. It was disappointing that we weren’t at least given the bag of words representations. I also attempted to reduce the feature representation with latent Dirichlet allocation, but it didn’t perform well – possibly because I couldn’t get the correct word counts.

                Conclusion

                Overall, this was a fun competition. Despite minor issues with the data and not having enough time to do everything I wanted to do, it was a great learning experience. From reading the summaries by the other teams, it appears that other competitors enjoyed it too. As always, I highly recommend Kaggle competitions to beginners who are trying to learn more about the field of data science and predictive modelling, and to more experienced data scientists who want to improve their skills.

                Subscribe
                  -
                  Alternatively, subscribe to RSS feed.

                  Public comments are closed, but I love hearing from readers. Feel free to +

                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/10/23/what-is-data-science/index.html b/2014/10/23/what-is-data-science/index.html index 1fa77d2c2..ae5600455 100644 --- a/2014/10/23/what-is-data-science/index.html +++ b/2014/10/23/what-is-data-science/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2014/10/23/what-is-data-science/data-skill-continuum.png 981w," src=https://yanirseroussi.com/2014/10/23/what-is-data-science/data-skill-continuum_hude8f4ba53ab678a51f562b1a637a59bc_5172_800x0_resize_box_3.png alt="Data skill continuum" loading=lazy>

                  This continuum contains two additional roles, which are often confused with data scientists:

                  • Data engineer: a software engineer that deals with data plumbing (traditional database setup, Hadoop, Spark and all the rest)
                  • Data analyst: a person who digs into data to surface insights, but lacks the skills to do so at scale (e.g., they know how to use Excel, Tableau and SQL but can’t build a web app from scratch)

                  Data science mixes all these roles. Because of this, there are few true data science positions for people with no work experience. A successful data scientist needs to be able to “become one with the data” by exploring it and applying rigorous statistical analysis (right-hand side of the continuum). But good data scientists also understand what it takes to deploy production systems, and are ready to get their hands dirty by writing code that cleans up the data or performs core system functionality (left-hand side of the continuum). Gaining all these skills takes time. It is still somewhat rare to find people who are true data scientists according to this definition, which is why Ofer Mendelevitch’s post recommends building teams that consist of people with skills from both sides of the continuum.

                  How is data science different from just science?

                  Data is everywhere. Extracting knowledge from data is an essential part of any science. Hence, the name data science doesn’t really capture what’s new about the field. The way I see it, the novelty of data science comes from the application of software to model any type of data in a way that generalises across domains. So while a physicist may use software to build models based on data, they won’t become a data scientist until they’ve gone and applied these skills to other fields (as many physicists end up doing). As Kaggle shows, data scientists can work on a wide variety of problems – from biology and physics to marketing, text mining and web search personalisation. It’s often the case in Kaggle competitions that the same people apply similar techniques to very different problems, obtaining results that significantly improve on the state of the art.

                  However, domain experts such as physicists aren’t going to be made redundant any time soon. Contrary to what Kaggle may have you believe, there is much more to data science than predictive modelling on a well-defined problem. Data scientists typically spend much of their time working with domain experts to define the problem, and chasing down diverse data sources to extract features that enable predictive modelling (also known as “the fun part”). Despite the existence of these less-glamorous aspects of data science, there’s still a lot of fun to be had working in the area. I highly recommend getting into data science to people who enjoy such challenges.

                  Getting started as a data scientist is actually pretty simple: become a software engineer, become a data analyst, learn how to model data using software (e.g., by participating in Kaggle competitions), and find a job as a data scientist. Obviously, it’s not going to happen overnight. It took me around 10 ten years, and I’m still learning.

                  Subscribe
                    -
                    Alternatively, subscribe to RSS feed.

                    Public comments are closed, but I love hearing from readers. Feel free to +

                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                    Thomas Packer (@ThomasPacker)
                    2015-05-18 17:53:00

                    So true. Thanks for saying it so well.
                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/11/05/bcrecommender-traction-update/index.html b/2014/11/05/bcrecommender-traction-update/index.html index 9778ed6f8..80567123e 100644 --- a/2014/11/05/bcrecommender-traction-update/index.html +++ b/2014/11/05/bcrecommender-traction-update/index.html @@ -2,7 +2,7 @@

                    BCRecommender Traction Update

                    November 5, 2014

                    This is the fifth part of a series of posts on my Bandcamp recommendations (BCRecommender) project. Check out previous posts on the general motivation behind this project, the system’s architecture, the recommendation algorithms, and initial traction planning.

                    In a previous post, I discussed my plans to apply the Bullseye framework from the Traction Book to BCRecommender, my Bandcamp recommendations project. In that post, I reviewed the 19 traction channels described in the book, and decided to focus on the three most promising ones: blogger outreach, search engine optimisation (SEO), and content marketing. This post discusses my progress to date.

                    Goals

                    My initial traction goals were rather modest: get some feedback from real people, build up steady nonzero traffic to the site, and then increase that traffic to 10+ unique visitors per day. It’s worth noting that I have four other main areas of focus at the moment, so BCRecommender is not getting all the attention I could potentially give it. Nonetheless, I have made good progress on achieving my goals (first two have been obtained, but traffic still fluctuates), and learnt a lot in the process.

                    Things that worked

                    Blogger outreach. The most obvious people to contact are existing Bandcamp fans. It was straightforward to generate a list of prolific fans with blogs, as Bandcamp allows people to populate their profile with a short bio and links to their sites. I worked my way through part of the list, sending each fan an email introducing BCRecommender and asking for their feedback. Each email required some manual work, as the vast majority of people don’t have their email address listed on their Bandcamp profile page. I was careful not to be too spammy, which seemed to work: about 50% of the people I contacted visited BCRecommender, 20% responded with positive feedback, and 10% linked to BCRecommender in some form, with the largest volume of traffic coming from my Hypebot guest post. The problem with this approach is that it doesn’t scale, but the most valuable thing I got out of it was that people like the project and that there’s a real need for it.

                    Twitter. I’m not sure where Twitter falls as a traction channel. It’s probably somewhere between (micro)blogger outreach and content marketing. However you categorise Twitter, it has been working well as a source of traffic. Simply finding people who may be interested in BCRecommender and tweeting related content has proven to be a rather low-effort way of getting attention, which is great at this stage. I have a few ideas for driving more traffic from Twitter, which I will try as I go.

                    Things that didn’t work

                    Content marketing. I haven’t really spent time doing serious content marketing apart from the Spotlights pilot. My vision for the spotlights was to generate quality articles automatically and showcase music on Bandcamp in an engaging way that helps people discover new artists, even if they don’t have a fan account. However, full automation of the spotlight feature would require a lot of work, and I think that there are lower-hanging fruits that I should focus on first. For example, finding interesting insights in the data and presenting them in an engaging way may be a better content strategy, as it would be unique to BCRecommender. For the spotlights, partnering with bloggers to write the articles may be a better approach than automation.

                    SEO. I expected BCRecommender to rank higher for “bandcamp recommendations” by now, as a result of my blogger outreach efforts. At the moment, it’s still on the second page for this query on Google, though it’s the first result on Bing and DuckDuckGo. Obviously, “bandcamp recommendations” is not the only query worth ranking for, but it’s very relevant to BCRecommender, and not too competitive (half of the first page results are old forum posts). One encouraging outcome from the work done so far is that my Hypebot guest post does appear on the first page. Nonetheless, I’m still interested in getting more search engine traffic. Ranking higher would probably require adding more relevant content on the site and getting more quality links (basically what SEO is all about).

                    Points to improve and next steps

                    I could definitely do better work on all of the above channels. Contrary to what’s suggested by the Bullseye framework, I would like to put more effort into the channels that didn’t work well. The reason is that I think they didn’t work well because of lack of attention and weak experiments, rather than due to their unsuitability to BCRecommender.

                    As mentioned above, my main limiting factor is a lack of time to spend on the project. However, there’s no pressing need to hit certain traction milestones by a specific deadline. My stretch goals are to get all Bandcamp fans to check out the project (hundreds of thousands of people), and have a significant portion of them convert by signing up to updates (tens of thousands of people). Getting there will take time. So far I’m finding the process educational and enjoyable, which is a pleasant surprise.

                    Subscribe
                      -
                      Alternatively, subscribe to RSS feed.

                      Public comments are closed, but I love hearing from readers. Feel free to +

                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/index.html b/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/index.html index d701c3790..7914f43bd 100644 --- a/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/index.html +++ b/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/gradient-boosting-out-of-bag-experiment-toy-dataset.png 858w," src=https://yanirseroussi.com/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/gradient-boosting-out-of-bag-experiment-toy-dataset_hu02dc1ebe47af12a7ec8f5877429b5dec_71277_800x0_resize_box_3.png alt="Gradient Boosting out of bag experiment (toy dataset)" loading=lazy>

                      These results are pretty cool, but this is still just a toy dataset (though repeating the experiment with 100 different random seeds to generate different toy datasets yields similar results). The next steps would be to repeat Ridgeway’s experiments from the GBM guide on multiple datasets to see whether the results generalise well, which will be the topic of a different post. Regardless of the final outcomes, this story illustrates the unexpected paths in which a Kaggle competition can take you. No matter what rank you end up obtaining and regardless of your skill level, there’s always something new to learn.

                      Update: I ran Ridgway’s experiments. The results are discussed in Stochastic Gradient Boosting: Choosing the Best Number of Iterations.

                      Subscribe
                        -
                        Alternatively, subscribe to RSS feed.

                        Public comments are closed, but I love hearing from readers. Feel free to +

                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/12/15/seo-mostly-about-showing-up/index.html b/2014/12/15/seo-mostly-about-showing-up/index.html index 5760ffdad..a3e0b8777 100644 --- a/2014/12/15/seo-mostly-about-showing-up/index.html +++ b/2014/12/15/seo-mostly-about-showing-up/index.html @@ -2,7 +2,7 @@

                        SEO: Mostly about showing up?

                        December 15, 2014

                        In previous posts about getting traction for my Bandcamp recommendations project (BCRecommender), I mentioned search engine optimisation (SEO) as one of the promising traction channels. Unfortunately, early efforts yielded negligible traffic – most new visitors came from referrals from blogs and Twitter. It turns out that the problem was not showing up for the SEO game: most of BCRecommender’s pages were blocked for crawling via robots.txt because I was worried that search engines (=Google) would penalise the website for thin/duplicate content.

                        Recently, I beefed up most of the pages, created a sitemap, and removed most pages from robots.txt. This resulted in a significant increase in traffic, as illustrated by the above graph. The number of organic impressions went up from less than ten per day to over a thousand. This is expected to go up even further, as only about 10% of pages are indexed. In addition, some traffic went to my staging site because it wasn’t blocked from crawling (I had to set up a new staging site that is password-protected and add a redirect from the old site to the production site – a bit annoying but I couldn’t find a better solution).

                        I hope Google won’t suddenly decide that BCRecommender content is not valuable or too thin. The content is automatically generated, which is “bad”, but it doesn’t “consist of paragraphs of random text that make no sense to the reader but which may contain search keywords”. As a (completely unbiased) user, I think it is valuable to find similar albums when searching for an album you like – an example that represents the majority of people that click through to BCRecommender. Judging from the main engagement measure I’m using (time spent on site), a good number of these people are happy with what they find.

                        More updates to come in the future. For now, my conclusion is: thin content is better than no content, as long as it’s relevant to what people are searching for and provides real value.

                        Subscribe
                          -
                          Alternatively, subscribe to RSS feed.

                          Public comments are closed, but I love hearing from readers. Feel free to +

                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/index.html b/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/index.html index efdba0b2e..d273a35f3 100644 --- a/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/index.html +++ b/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/index.html @@ -9,7 +9,7 @@ https://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/gradient-boosting-out-of-bag-experiments-uci-datasets.png 591w," src=https://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/gradient-boosting-out-of-bag-experiments-uci-datasets.png alt="Gradient Boosting out of bag experiment (UCI datasets)" loading=lazy>

                          The following table shows the raw data that was used to produce the figure.

                          DatasetCVSKOTSO
                          creditrating0.99620.97711
                          breastcancer10.66750.4869
                          mushrooms0.95880.99631
                          abalone10.97540.9963
                          ionosphere0.991910.8129
                          diabetes10.98690.9985
                          autoprices10.95650.5839
                          autompg10.87530.9948
                          bostonhousing10.82990.5412
                          haberman10.97930.9266
                          cpuperformance0.99340.91601
                          adult10.98240.9991

                          The main finding is that CV remains the most reliable approach. Even when CV is not the best-performing method, it’s not much worse than the best method (this is in line with Ridgeway’s findings). TSO yielded the best results on 3/12 of the datasets, and beat SKO 7/12 times. However, TSO’s results are the most variant of the three methods: when it fails, it often yields very poor results.

                          In conclusion, stick to cross-validation for the best results. It’s more computationally intensive than SKO and TSO, but can be parallelised. I still think that there may be a way to avoid cross-validation, perhaps by extending SKO/TSO in more intelligent ways (see some interesting ideas by Eugene Dubossarsky here and here). Any comments/ideas are very welcome.

                          Subscribe
                            -
                            Alternatively, subscribe to RSS feed.

                            Public comments are closed, but I love hearing from readers. Feel free to +

                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/01/15/automating-parse-com-bulk-data-imports/index.html b/2015/01/15/automating-parse-com-bulk-data-imports/index.html index 32088aca6..2ce64808b 100644 --- a/2015/01/15/automating-parse-com-bulk-data-imports/index.html +++ b/2015/01/15/automating-parse-com-bulk-data-imports/index.html @@ -4,7 +4,7 @@

                            See the script’s source for a detailed explanation of the command-line arguments.

                            It is worth noting that the script doesn’t do any post-upload verification on the collection. This is done by an extra bit of Python code that verifies that the collection has the expected number of objects, and tries to query the collection sorted by all the keys that are supposed to be indexed (for large collections, it takes Parse a while to index all the fields, which may result in timeouts). Once these conditions are fulfilled, the Parse hosting code is updated to point to the new collection. For security, I added a bot user that has access only to the Parse app that it needs to update. Unlike the root user, this bot user can’t delete the app. As the config file contains the bot’s password, it should be encrypted and stored in a safe place (like the Parse master key).

                            That’s it! I hope that other people would find this solution useful. Any suggestions/comments/issues are very welcome.


                            Image source: Parse Blog.

                            Subscribe
                              -
                              Alternatively, subscribe to RSS feed.

                              Public comments are closed, but I love hearing from readers. Feel free to +

                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                              Walter
                              2015-07-30 08:22:30

                              Hi, very nice trick! Trying to implement this as we speak, does this code still work? I get to the collections page, but I don’t think the upload is working. I’m new to Phantomjs. Thanks!

                              Yanir Seroussi
                              2015-07-30 21:20:45

                              Hi Walter! Yeah, the code stopped working when Parse redesigned their website. I never fixed it because I ended up porting my projects away from Parse. If you fix it let me know and I’ll update this post. By the way, you may find it easier to use Selenium (or something similar) as a wrapper around PhantomJS, as it should result in cleaner code. For example, check out Python’s Selenium bindings: http://selenium.googlecode.com/svn/trunk/docs/api/py/index.html
                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | diff --git a/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/index.html b/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/index.html index 6f4a16a2d..50713f682 100644 --- a/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/index.html +++ b/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/index.html @@ -5,7 +5,7 @@

                              These records describe the session (SessionID = 744899) of the user with USERID 123123123, performed on the 23rd day of the dataset. The user submitted the query with QUERYID 192902, which contains terms with TermIDs 4857,3847,2939. The URL with URLID 632428 placed on the domain DomainID 2384 is the top result on the corresponding SERP. 1403 units of time after beginning of the session the user clicked on the result with URLID 632428 (ranked first in the list).

                              While this may seem daunting at first, the data is actually quite simple. For each search session, we know the user, the queries they’ve made, which URLs and domains were returned in the SERP (search engine result page), which results they’ve clicked, and at what point in time the queries and clicks happened.

                              Goal and evaluation

                              The goal of the competition is to rerank the results in each SERP such that the highest-ranking documents are those that the user would find most relevant. As the name of the competition suggests, personalising the results is key, but non-personalised approaches were also welcome (and actually worked quite well).

                              One question that arises is how to tell from this data which results the user finds relevant. In this competition, the results were labelled as either irrelevant (0), relevant (1), or highly relevant (2). Relevance is a function of clicks and dwell time, where dwell time is the time spent on the result (determined by the time that passed until the next query or click). Irrelevant results are ones that weren’t clicked, or those for which the dwell time is less than 50 (the time unit is left unspecified). Relevant results are those that were clicked and have dwell time of 50 to 399. Highly relevant results have dwell time of at least 400, or were clicked as the last action in the session (i.e., it is assumed the user finished the session satisfied with the results rather than left because they couldn’t find what they were looking for).

                              This approach to determining relevance has some obvious flaws, but it apparently correlates well with actual user satisfaction with search results.

                              Given the above definition of relevance, one can quantify how well a reranking method improves the relevance of the results. For this competition, the organisers chose the normalised discounted cumulative gain (NDCG) measure, which is a fancy name for a measure that, in the words of Wikipedia, encodes the assumptions that:

                              • Highly relevant documents are more useful when appearing earlier in a search engine result list (have higher ranks)
                              • Highly relevant documents are more useful than marginally relevant documents, which are in turn more useful than irrelevant documents.

                              SEO insights and other thoughts

                              A key insight that is relevant to SEO and privacy, is that even without considering browser-based tracking and tools like Google Analytics (which may or may not be used by Google to rerank search results), search engines can infer a lot about user behaviour on other sites, just based on user interaction with the SERP. So if your users bounce quickly because your website is slow to load or ranks highly for irrelevant queries, the search engine can know that, and will probably penalise you accordingly.

                              This works both ways, though, and is evident even on search engines that don’t track personal information. Just try searching for “f” or “fa” or “fac” using DuckDuckGo, Google, Bing, Yahoo, or even Yandex. Facebook will be one of the top results (most often the first one), probably just because people tend to search for or visit Facebook after searching for one of those terms by mistake. So if your website ranks poorly for a term for which it should rank well, and your users behave accordingly (because, for example, they’re searching for your website specifically), you may magically end up with better ranking without any changes to inbound links or to your site.

                              Another thing that is demonstrated by this competition’s dataset is just how much data search engines consider when determining ranking. The dataset is just a sample of logs for one city for one month. I don’t like throwing the words “big data” around, but the full volume of data is pretty big. Too big for anyone to grasp and fully understand how exactly search engines work, and this includes the people who build them. What’s worth keeping in mind is that for all major search engines, the user is the product that they sell to advertisers, so keeping the users happy is key. Any changes made to the underlying algorithms are usually done with the end-user in mind, because not making such changes may kill the search engine (remember AltaVista?). Further, personalisation means that different users see different results for the same query. So my feeling is that it’s somewhat futile to do any SEO beyond making the website understandable by search engines, acquiring legitimate links, and just building a website that people would want to visit.

                              Next steps

                              With those thoughts out of the way, it’s time to describe the way we addressed the challenge. This is covered in the next post, Learning to rank for personalised search.

                              Subscribe
                                -
                                Alternatively, subscribe to RSS feed.

                                Public comments are closed, but I love hearing from readers. Feel free to +

                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/index.html b/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/index.html index 042d4d712..0ab7c3bab 100644 --- a/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/index.html +++ b/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/index.html @@ -2,7 +2,7 @@

                                Learning to rank for personalised search (Yandex Search Personalisation – Kaggle Competition Summary – Part 2)

                                February 11, 2015

                                This is the second and last post summarising my team’s solution for the Yandex search personalisation Kaggle competition. See the first post for a summary of the dataset, evaluation approach, and some thoughts about search engine optimisation and privacy. This post discusses the algorithms and features we used.

                                To quickly recap the first post, Yandex released a 16GB dataset of query & click logs. The goal of the competition was to use this data to rerank query results such that the more relevant results appear before less relevant results. Relevance is determined by time spent on each clicked result (non-clicked results are deemed irrelevant), and overall performance is scored using the normalised discounted cumulative gain (NDCG) measure. No data about the content of sites or queries was given – each query in the dataset is a list of token IDs and each result is a (url ID, domain ID) pair.

                                First steps: memory-based heuristics

                                My initial approach wasn’t very exciting: it involved iterating through the data, summarising it in one way or another, and assigning new relevance scores to each (user, session, query) combination. In this early stage I also implemented an offline validation framework, which is an important part of every Kaggle competition: in this case I simply set aside the last three days of data for local testing, because the test dataset that was used for the leaderboard consisted of three days of log data.

                                Somewhat surprisingly, my heuristics worked quite well and put me in a top-10 position on the leaderboard. It seems like the barrier of entry for this competition was higher than for other Kaggle competitions due to the size of the data and the fact that it wasn’t given as preprocessed feature vectors. This was evident from questions on the forum, where people noted that they were having trouble downloading and looking at the data.

                                The heuristic models that worked well included:

                                • Reranking based on mean relevance (this just swapped positions 9 & 10, probably because users are more likely to click the last result)
                                • Reranking based on mean relevance for (query, url) and (query, domain) pairs (non-personalised improvements)
                                • Downranking urls observed previously in a session

                                Each one of the heuristic models was set to output relevance scores. The models were then ensembled by simply summing the relevance scores.

                                Then, I started playing with a collaborative-filtering-inspired matrix factorisation model for predicting relevance, which didn’t work too well. At around that time, I got too busy with other stuff and decided to quit while I’m ahead.

                                Getting more serious with some team work and LambdaMART

                                A few weeks after quitting, I somehow volunteered to organise Kaggle teams for newbies at the Sydney Data Science Meetup group. At that point I was joined by my teammates, which served as a good motivation to do more stuff.

                                The first thing we tried was another heuristic model I read about in one of the papers suggested by the organisers: just reranking based on the fact that people often repeat queries as a navigational aid (e.g., search for Facebook and click Facebook). Combined in a simple linear model with the other heuristics, this put us at #4. Too easy 🙂

                                With all the new motivation, it was time to read more papers and start doing things properly. We ended up using Ranklib’s LambdaMART implementation as one of our main models, and also used LambdaMART to combine the various models (the old heuristics still helped the overall score, as did the matrix factorisation model).

                                Using LambdaMART made it possible to directly optimise the NDCG measure, turning the key problem into feature engineering, i.e., finding good features to feed into the model. Explaining how LambdaMART works is beyond the scope of this post (see this paper for an in-depth discussion), but the basic idea (which is also shared by other learning to rank algorithms) is that rather than trying to solve the hard problem of predicting relevance (i.e., a regression problem), the algorithm tries to predict the ranking that yields the best results according to a user-chosen measure.

                                We tried many features for the LambdaMART model, but after feature selection (using a method learned from Phil Brierley’s talk) the best features turned out to be:

                                • percentage_recurrent_term_ids: percentage of term IDs from the test query that appeared previously in the session — indicates if this query refines previous queries
                                • query_mean_ndcg: historical NDCG for this query — indicates how satisfied people are with the results of this query. Interestingly, we also tried query click entropy, but it performed worse. Probably because we’re optimising the NDCG rather than click-through rate.
                                • query_num_unique_serps: how many different result pages were shown for this query
                                • query_mean_result_dwell_time: how much time on average people spend per result for this query
                                • user_mean_ndcg: like query_mean_ndcg, but for users — a low NDCG indicates that this user is likely to be dissatisfied with the results. As for query_mean_ndcg, adding this feature yielded better results than using the user’s click entropy.
                                • user_num_click_actions_with_relevance_0: over the history of this user, how many of their clicks had relevance 0 (i.e., short dwell time). Interestingly, user_num_click_actions_with_relevance_1 and user_num_click_actions_with_relevance_2 were found to be less useful.
                                • user_num_query_actions: number of queries performed by the user
                                • rank: the original rank, as assigned by Yandex
                                • previous_query_url_relevance_in_session: modelling repeated results within a session, e.g., if a (query, url) pair was already found irrelevant in this session, the user may not want to see it again
                                • previous_url_relevance_in_session: the same as previous_query_url_relevance_in_session, but for a url regardless of the query
                                • user_query_url_relevance_sum: over the entire history of the user, not just the session
                                • user_normalised_rank_relevance: how relevant does the user usually find this rank? The idea is that some people are more likely to go through all the results than others
                                • query_url_click_probability: estimated simply as num_query_url_clicks / num_query_url_occurrences (across all the users)
                                • average_time_on_page: how much time people spend on this url on average

                                Our best submission ended up placing us at the 9th place (out of 194 teams), which is respectable. Things got a bit more interesting towards the end of the competition – if we had used the original heuristic model that put at #4 early on, we would have finished 18th.

                                Conclusion

                                I really enjoyed this competition. The data was well-organised and well-defined, which is not something you get in every competition (or in “real life”). Its size did present some challenges, but we stuck to using flat files and some preprocessing and other tricks to speed things up (e.g., I got to use Cython for the first time). It was good to learn how learning to rank algorithms work and get some insights on search personalisation. As is often the case with Kaggle competitions, this was time well spent.

                                Subscribe
                                  -
                                  Alternatively, subscribe to RSS feed.

                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                  nitin
                                  2017-12-17 13:02:37

                                  I do not understand how your featureset helped the model to learn anything. For example, user_num_query_actions: number of queries performed by the user

                                  How will it affect the order of search results for a new/test query.

                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | diff --git a/2015/03/22/the-long-road-to-a-lifestyle-business/index.html b/2015/03/22/the-long-road-to-a-lifestyle-business/index.html index b3a1aded0..5b0c6ffe1 100644 --- a/2015/03/22/the-long-road-to-a-lifestyle-business/index.html +++ b/2015/03/22/the-long-road-to-a-lifestyle-business/index.html @@ -2,7 +2,7 @@

                                  The long road to a lifestyle business

                                  March 22, 2015

                                  Almost a year ago, I left my last full-time job and decided to set on an independent path that includes data science consulting and work on my own projects. The ultimate goal is not to have to sell my time for money by generating enough passive income to live comfortably. My five main areas of focus are – in no particular order – personal branding & networking, data science contracting, Bandcamp Recommender, Price Dingo, and marine conservation. This post summarises what I’ve been doing in each of these five areas, including highlights and lowlights. So far, it’s way better than having a “real” job. I hope this post will help others who are on a similar journey (there seem to be more and more of us – I’d love to hear from you).

                                  Personal branding & networking

                                  Finding clients requires considerably more work than finding a full-time job. As with job hunting, the ideal situation is where people come to you for help, rather than you chasing them. To this end, I’ve been networking a lot, giving talks, writing up posts and working on distributing them. It may be harder than getting a full-time job, but it’s also much more interesting.

                                  Highlights: going viral in China, getting a post featured in KDNuggets
                                  Lowlights: not having enough time to write all the things and meet all the people

                                  Data science contracting

                                  My goal with contracting/consulting is to have a steady income stream while working on my own projects. As my projects are small enough to be done only by me (with optional outsourcing to contractors), this means I have infinite runway to pursue them. While this is probably not the best way of building a Silicon Valley-style startup that is going to make the world a better place, many others have applied this approach to building a so-called lifestyle business, which is what I want to achieve.

                                  Early on, I realised that doing full-on consulting would be too time consuming, as many clients expect full-time availability. In addition, constantly needing to find new clients means that not much time would be left for work on my own projects. What I really wanted was a stable part-time gig. The first one was with GetUp (who reached out to me following a workshop I gave at General Assembly), where I did some work on forecasting engagement and churn. In parallel, I went through the interview process at DuckDuckGo, which included delivering a piece of work to production. DuckDuckGo ended up wanting me to work full-time (like a few other companies), so last month I started a part-time (three days a week) contract at Commonwealth Bank. I joined a team of very strong data scientists – it looks like it’s going to be interesting.

                                  Highlights: seeing my DuckDuckGo work every time I search for a Python package, the work environment at GetUp
                                  Lowlights: chasing leads that never eventuated

                                  Bandcamp Recommender (BCRecommender)

                                  I’ve written a several posts about BCRecommender, my Bandcamp music recommendation project. While I’ve always treated it as a side-project, it’s been useful in learning how to get traction for a product. It now has thousands of monthly users, and is still growing. My goal for BCRecommender has changed from the original one of finding music for myself to growing it enough to be a noticeable source of traffic for Bandcamp, thereby helping artists and fans. Doing it in side-project mode can be a bit challenging at times (because I have so many other things to do and a long list of ideas to make the app better), but I’ve been making gradual progress and discovering a lot of great music in the process.

                                  Highlights: every time someone gives me positive feedback, every time I listen to music I found using BCRecommender
                                  Lowlights: dealing with Parse issues and random errors

                                  Price Dingo

                                  The inability to reliably compare prices for many types of products has been bothering me for a while. Unlike general web search, where the main providers rank results by relevance, most Australian price comparison engines still require merchants to pay to even have their products listed. This creates an obvious bias in the results. To address this bias, I created Price Dingo – a user-centric price comparison engine. It serves users with results they can trust by not requiring merchants to pay to have their products listed. Just like general web search engines, the main ranking factor is relevancy to the user. This relevancy is also achieved by implementing Price Dingo as a network of independent sites, each focused on a specific product category, with the first category being scuba diving gear.

                                  Implementing Price Dingo hasn’t been too hard – the main challenge has been finding the time to do it with all the other stuff I’ve been doing. There are still plenty of improvements to be made to the site, but now the main goal is to get enough traction to make ongoing time investment worthwhile. Judging by the experience of Booko’s founder, there is space in the market for niche price comparison sites and apps, so it is just a matter of execution.

                                  Highlights: being able to finally compare dive gear prices, the joys of integrating Algolia
                                  Lowlights: extracting data from messy websites – I’ve seen some horrible things…

                                  Marine conservation

                                  The first thing I did after leaving my last job was go overseas for five weeks, which included a ten-day visit to Israel (rockets!) and three weeks of conservation diving with New Heaven Dive School in Thailand. Back in Sydney, I joined the Underwater Research Group of NSW, a dive club that’s involved in many marine conservation and research activities, including Reef Life Survey (RLS) and underwater cleanups. With URG, I’ve been diving more than before, and for a change, some of my dives actually do good. I’d love to do this kind of stuff full-time, but there’s a lot less money in getting people to do less stuff (i.e., conservation and sustainability) than in consuming more. The compromise for now is that a portion of Price Dingo’s scuba revenue goes to the Australian Marine Conservation Society, and the plan is to expand this to other charities as more categories are added. Update – May 2015: I decided that this compromise isn’t good enough for me, so I shut down Price Dingo to focus on projects that are more aligned with my values.

                                  Highlights: becoming a certified RLS diver, pretty much every dive
                                  Lowlights: cutting my hand open by falling on rocks on the first day of diving in Thailand

                                  The future

                                  So far, I’m pretty happy with this not-having-a-job-doing-my-own-thing business. According to The 1000 Day Rule, I still have a long way to go until I get the lifestyle I want. It may even take longer than 1000 days given my decision to not work full-time on a single profitable project, together with my tendency to take more time off than I would if I had a “real” job. But the beauty of this path is that there are no investors breathing down my neck or the feeling of mental rot that comes with a full-time job, so there’s really no rush and I can just enjoy the ride.

                                  Subscribe
                                    -
                                    Alternatively, subscribe to RSS feed.

                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                    Ralph Haygood
                                    2016-01-29 06:08:15

                                    “What I really wanted was a stable part-time gig.”: They’re remarkably hard to find. It’s an absurdity of our time that many people are overemployed - selling more of their time than they want for more money than they need - even while many other people are underemployed - unable to sell enough of their time for enough money to live comfortably.

                                    Yanir Seroussi
                                    2016-01-30 09:27:23

                                    That’s very true. The interesting thing is that it’s a problem that is not unique to this century. It was discussed by Thoreau in Walden (1854), Bertrand Russell in In Praise of Idleness (1932), and David Graeber in On the Phenomenon of Bullshit Jobs (2013), to name a few. People seem to be worried about robots taking their jobs, but the scarier thought is that robots will never take our jobs, because we’ll keep coming up with ways of staying employed rather than enjoy the affluence afforded by technological advancements.
                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/04/24/my-divestment-from-fossil-fuels/index.html b/2015/04/24/my-divestment-from-fossil-fuels/index.html index 98388aa1f..eacdd8f01 100644 --- a/2015/04/24/my-divestment-from-fossil-fuels/index.html +++ b/2015/04/24/my-divestment-from-fossil-fuels/index.html @@ -2,7 +2,7 @@

                                    My divestment from fossil fuels

                                    April 24, 2015

                                    This post covers recent choices I've made to reduce my exposure to fossil fuels, including practical steps that can be taken by Australians and generally applicable lessons.

                                    I recently read Naomi Klein’s This Changes Everything, which deeply influenced me. The book describes how the world has been dragging its feet when it comes to reducing carbon emissions, and how we are coming very close to a point where climate change is likely to spin out of control. While many of the facts presented in the book can be very depressing, one ray of light is that it is still not too late to act. There are still things we can do to avoid catastrophic climate change.

                                    One such thing is divestment from fossil fuels. Fossil fuel companies have committed to extracting (and therefore burning) more than what scientists agree is the safe amount of carbon that can be pumped into the atmosphere. While governments have been rather ineffective in stopping this (the current Australian government is even embarrassingly rolling back emission-reduction measures), divesting your money from such companies can help take away the social licence of these companies to do as they please. Further, this may be a smart investment strategy because the world is moving towards renewable energy. Indeed, according to one index, investors who divested from fossil fuels have had higher returns than conventional investors over the last five years.

                                    It’s worth noting that even if you disagree with the scientific consensus that releasing billions of tonnes of greenhouse gases into the atmosphere increases the likelihood of climate change, you should agree that it’d be better to stop breathing all the pollutants that result from burning fossil fuels. Further, the environmental damage that comes with extracting fossil fuels is something worth avoiding. Examples include the Deepwater Horizon oil spill, numerous cases of poisoned water due to fracking, and the potential damage to the Great Barrier Reef due to coal mine expansion. Even climate change deniers would admit that divestment from fossil fuels and a rapid move to clean renewables will prevent such disasters.

                                    The rest of this post describes steps I’ve recently taken towards divesting from fossil fuels. These are mostly relevant to Australians, though other countries may have similar options.

                                    Superannuation

                                    In Australia, we have compulsory superannuation (commonly known as super), meaning that most working Australians have some money invested somewhere. As this money is only available at retirement, investors can afford to optimise for long-term returns. Many super funds allow investors to choose what to invest in, and switching funds is relatively straightforward. My super fund is UniSuper. Last week, I switched my plan from Balanced, which includes investments in coal miners Rio Tinto and BHP Billiton, to 75% Sustainable Balanced, which doesn’t directly invest in fossil fuels, and 25% Global Environment Opportunities, which is focused on companies with a green agenda such as Tesla. This switch was very simple – I wish I had done it earlier. If you’re interested in making a similar switch, check out Superswitch’s guide to fossil-free super options.

                                    Energy

                                    While our previous energy retailer (ClickEnergy) isn’t one of the big three retailers who are actively lobbying the government to reduce the renewable energy target for 2020, my partner and I decided to switch to Powershop, as it appears to be the greenest energy retailer in New South Wales. Powershop supports maintaining the renewable energy target in its current form and provides free carbon offsets for all non-renewable energy. In addition, Powershop allows customers to purchase 100% green power from renewables – an option that we choose to take. With the savings from moving to Powershop and the extra payment for green power, our bill is expected to be more or less the same as before. Everyone wins!

                                    Note: If you live in New South Wales or Victoria and generally support what GetUp is doing, you can sign up via the links on this page, and GetUp will be paid a referral fee by Powershop.

                                    Banking

                                    There’s been a lot of focus recently on financing provided by the major banks to fossil fuel companies. The problem is that – unlike with super and energy – there aren’t many viable alternatives to the big banks. Reading the statements by smaller banks and credit unions, it is clear that they don’t provide financing to polluters just because they’re too small or not focused on commercial lending. Further, some of the smaller banks invest their money with the bigger banks. If the smaller banks were to become big due to the divestment movement, they may end up financing polluters. Unfortunately, changing your bank doesn’t give you more control over how your chosen financial institute uses your money.

                                    For now, I think it makes sense to push the banks to become fossil free by putting them on notice or participating in demonstrations. With enough pressure, one of the big banks may make a strong statement against lending to polluters, and then it’ll be time to act on the notices. One thing that the big banks care about is customer satisfaction and public image. Sending a strong message about the connection between financing polluters and satisfaction may be enough to make a difference. I’ll be tracking news in this area and will possibly make a switch in the future, depending on how things evolve.

                                    Transportation

                                    My top transportation choices are cycling and public transport, followed by driving when the former two are highly inconvenient (e.g., when going scuba diving). Every bike ride means less pollution and is a vote against fossil fuels. Further, bike riding is my main form of exercise, so I don’t need to set aside time to go to the gym. Finally, it’s almost free, and it’s also the fastest way of getting to the city from where I live.

                                    Since January, I’ve been allowing people to borrow my car through Car Next Door. This service, which is currently active in Sydney and Melbourne, allows people to hire their neighbours’ cars, thereby reducing the number of cars on the road. They also carbon offset all the rides taken through the service. While making my car available has made using it slightly less convenient (because I need to book it for myself), it’s also saved me money, so far covering the cost of insurance and roadside assistance. With my car sitting idle for 95% of the time before joining Car Next Door, it’s definitely another win-win situation. If you’d like to join Car Next Door as either a borrower or an owner, you can use this link to get $15 credit.

                                    Other areas and next steps

                                    Many of the choices we make every day have the power to reduce energy demand. These choices often make our life better, as seen with the bike riding example above. There’s a lot of material online about these green choices, which I may cover from my angle in another post. In general, I’m planning to be more active in the area of environmentalism. While this may come at the cost of reduced focus on my other activities, I would rather be more a part of the solution than a part of the problem. I’ll update as I go – please subscribe to get notified when updates occur.

                                    Subscribe
                                      -
                                      Alternatively, subscribe to RSS feed.

                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                      Alex
                                      2018-12-25 13:44:02

                                      Thanks for sharing your standpoint on this.
                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/index.html b/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/index.html index 738f8ebfa..081c56642 100644 --- a/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/index.html +++ b/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/index.html @@ -2,7 +2,7 @@

                                      First steps in data science: author-aware sentiment analysis

                                      May 2, 2015

                                      People often ask me what’s the best way of becoming a data scientist. The way I got there was by first becoming a software engineer and then doing a PhD in what was essentially data science (before it became such a popular term). This post describes my first steps in the field with the goal of helping others who are interested in making the transition from pure software engineering to data science.

                                      While my first steps were in a PhD program, I don’t think that going through the formal PhD process is necessary if you wish to become a data scientist. Self-motivated individuals can get very far by making use of the abundance of learning resources available online. In fact, one can make progress much faster than in a PhD, because PhD programs have many overheads.

                                      This post is organised as a list of steps. Despite the sequential numbering, many steps can be done in parallel. These steps roughly recount the work I’ve done to publish my first paper, which was co-authored by Ingrid Zukerman and Fabian Bohnert. Most of the technical details are intentionally omitted. Readers who are interested in learning more are invited to read the original paper or chapter 6 in my thesis, which includes more thorough experiments and explanations.

                                      Step one: Find a problem to work on

                                      Even if you know nothing about the machine learning and statistics side of data science, it’s important to find a problem to work on. Ideally it’d be something you find personally interesting, as this helps with motivation. You could use a predefined problem such as a Kaggle competition or one of the UCI datasets. Alternatively, you could collect the data yourself to make things a bit more challenging.

                                      In my case, I was interested in natural language processing and user modelling. My supervisor was given a grant to work on sentiment analysis of opinion polls, which was my first direction of research. This quickly changed to focus on the connection between authors and the way they express their sentiments, with the application of harnessing this connection to improve the accuracy of sentiment analysis algorithms. For the purpose of this research, I collected a dataset of texts by the most prolific IMDb users. The problem was to infer the ratings these users assigned to their own reviews, with the hypothesis that methods that take author identity into account would outperform methods that ignore authorship information.

                                      Step two: Close your knowledge gaps

                                      Whatever problem you choose, you will have some knowledge gaps that require filling. Wikipedia, textbooks, and online courses will be your best guide for foundational areas like machine learning and statistics. Reading academic papers is often required to get a better understanding of recent work on the specific problem you’re trying to solve.

                                      Doing a PhD afforded me the luxury of spending about a month just reading papers. Most of the ~200 papers I read were on sentiment analysis, which gave me a good overview of what’s been done in the field. However, the best thing I’ve done was to stop reading and move on to working on the problem. This is also the best advice I can give: there’s no better way to learn than getting your hands dirty working on a problem.

                                      Step three: Get your hands dirty

                                      With a well-defined problem and the knowledge gaps more-or-less closed, it is time to come up with a plan and implement it. Due to my background in software engineering and some exposure to early collaborative filtering approaches to recommender systems, my plan was very much a part of what Leo Breiman called the algorithmic modelling culture. That is, I was more focused on developing algorithms that work than on modelling the process that generated the data. This approach is arguably more in line with the mindset that software engineers tend to have than with the approach of mathematicians and statisticians.

                                      The plan was quite simple:

                                      • Reproduce results that showed that rating inference models trained on enough texts by the target author (i.e., the author who wrote the text whose rating we want to predict) outperform models trained on texts by multiple authors
                                      • Use an approach inspired by collaborative filtering to combine multiple single-author models to infer ratings for texts by the target author, where those models are weighted by similarity to the target author
                                      • Experiment with multiple similarity measurements under various constraints on the number of texts available by the training and target authors
                                      • Iterate on these ideas until the results are publishable

                                      The rationale behind this plan was that while different people express their sentiments differently, similar people would express their sentiments similarly (e.g., use of understatements varies by culture). The key motivation was Pang and Lee’s finding that a model trained on a single author is best if we have enough texts by this author.

                                      The way I implemented the plan was vastly different from how I’d do it today. This was 2009, and using Java with the Weka package for the core modelling seemed like a huge improvement over the C/C++ I was used to. I relied heavily on the university grid to run experiments and wrote a bunch of code to handle experimental logic, including some Perl scripts for post-processing. It ended up being pretty messy, but it worked and I got publishable results. If I were to do the same work today, I’d use Python for everything. IPython Notebook is a great way of keeping track of experimental work, and Python packages like pandas, scikit-learn, gensim, TextBlob, etc. are mature and easy to use for data science applications.

                                      Step four: Publish your results

                                      Having a deadline for publishing results can be stressful, but it has two positive outcomes. First, making your work public allows you to obtain valuable feedback. Second, hard deadlines are great in making you work towards a tangible goal. You can always keep iterating to get infinitesimal improvements, but publication deadlines force you to decide that you’ve done enough.

                                      In my case, the deadline for the UMAP 2010 conference and the promise of a free trip to Hawaii served as excellent motivators. But even if you don’t have the time or energy to get an academic paper published, you should set yourself a deadline to publish something on a blog or a forum, or even as a report to a mentor who can assess your work. Receiving continuous feedback is a key factor in improvement, so release early and release often.

                                      Step five: Improve results or move on

                                      Congratulations! You have published the results of your study. What now? You can either keep working on the same problem – try more approaches, add more data, change the constraints, etc. Or you can move on to work on other problems that interest you.

                                      In my case, I had to go back to iterate on the results of the first paper because of things I learned later. I ended up rerunning all the experiments to make things fit together into a more-or-less coherent story for the thesis (writing a thesis is one of the main overheads that comes with doing a PhD). If I had a choice, I wouldn’t have done that. I would instead have pursued more sensible enhancements to the work presented in the paper, such as using the author as a feature, employing more robust ensemble methods, and testing different base methods than support vector machines. Nonetheless, I still think that the core idea – that the identity of authors should be taken into account in sentiment analysis – is still relevant and viable today. But I’ve taken my own advice and moved on.

                                      Subscribe
                                        -
                                        Alternatively, subscribe to RSS feed.

                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                        Robert Klein (@PaperbackLegacy)
                                        2015-05-18 18:34:00

                                        Thanks for the stimulation. I’m still fascinated by the lure to extract sentiment from text, but it seems like so often the sentiment that the author intended never fully came to expression in the text. Maybe an interdisciplinary approach will be required to teach machines to parse the intentions implicit in text, and, like other media phenomena, a loop will have to form: perhaps the awareness that explicit intentions and sentiment are of benefit to authors in a world that (one day) automates the sorting of all its documents will cause writing styles to adapt. The effect of the best we can on what we’re doing now is one of those things you begin to see a pattern in. Here’s an API that correlates patterns of unstructured info: dev.keywordmeme.com Would love your feedback. Let me know if it’s useful to you or if you have any comments. Well done on the carbon post, btw. Glad I found your blog.

                                        Yanir Seroussi
                                        2015-05-22 08:49:02

                                        Thank you for the comment! I agree that analysing sentiment is very tricky due to the fact that people often don’t express themselves so well. If I remember correctly, inter-annotator agreement on some sentiment analysis tasks is only 70-80%, so it’s likely that we will ever have perfect performance by machines.

                                        dev.keywordmeme.com redirects to a github page – where is the API?

                                        Robert Klein (@PaperbackLegacy)
                                        2015-05-26 19:43:10

                                        You bet. I’m fascinated to see what seems to be a real live push toward an interdisciplinary approach. That 70-80% performance might be pushed over the hump by humans with special training until such a time as the process can be formalized. It looks like auditing ML-driven processes could be a new category of employment through this next technological plateau. The human-machine relationship in a friendly old configuration! Sorry about the link. This should work: http://www.keywordmeme.com/. It makes you register, just a heads up. Hit the engineers up on github if you have any questions or if things aren’t working. Which is possible. Take care! :)

                                        A
                                        2015-09-04 07:31:28

                                        Hi Yanir

                                        Thank you very much for this post. Helpul for somebody like me seeking to be a data scientist.

                                        I’m a software engineer, currently master data architect.

                                        I’m taking MOOCs in order to fill the gaps, so let’s say I’m on a good track :)

                                        However, once problem found and hands got dirty, how to find a mentor ? afterwards, get published ?

                                        I think this would be hard via academic

                                        Yanir Seroussi
                                        2015-09-04 08:46:51

                                        Finding a mentor depends on where you are. Good places to start would be your current workplace (if you work with data scientists), or local meetups (if there are any in your area). Another option would be to contribute to open source projects in the field as a way of getting to know people and getting feedback. Finally, there are courses like the one by Thinkful, where you can pay to be mentored.

                                        Regarding getting published, I agree that it’d be hard to get published in many academic venues without help from people who know how it’s done. However, you can always start your own blog and link to it from places like Reddit and DataTau. Even if you don’t get any feedback, publishing often forces you to think more deeply about the subject of your article.

                                        Amine
                                        2015-09-04 11:18:10

                                        At the workplace, it will be a bit hard.

                                        I Live in Paris, meetups would be a good option.

                                        You’re right, publishing forces to think more deeply, feedbacks from readers are also good way to learn.

                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2015/06/06/hopping-on-the-deep-learning-bandwagon/index.html b/2015/06/06/hopping-on-the-deep-learning-bandwagon/index.html index 9f5c4aec1..655504b7b 100644 --- a/2015/06/06/hopping-on-the-deep-learning-bandwagon/index.html +++ b/2015/06/06/hopping-on-the-deep-learning-bandwagon/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2015/06/06/hopping-on-the-deep-learning-bandwagon/bandcamp-album-covers-by-genre.png 828w," src=https://yanirseroussi.com/2015/06/06/hopping-on-the-deep-learning-bandwagon/bandcamp-album-covers-by-genre_hu98267f967d1b66bf7a519f3fa620b70e_1042875_800x0_resize_box_3.png alt="Bandcamp album covers by genre" loading=lazy>

                                        It is apparent that some genres can be inferred more easily than others, especially when browsing through the full dataset. For example, metal albums tend to be pretty distinct. I doubt that predictive accuracy would be very high, but I think that it can definitely be much better than the random baseline of 10%.

                                        For training, validation and testing I decided to use a static stratified 80%/10%/10% split of the dataset. It quickly became apparently that the full dataset is too big for development purposes, making it hard to quickly test code on my local machine. To address this, I created a local development dataset, using an 80%/10%/10% split of 1,000 images from the full training subset.

                                        The code for downloading the dataset and creating the splits is available from the project repository on GitHub. This repository will include all the code for the project as it evolves. I will try to keep it well-documented enough to be useful for others, though it assumes some familiarity with Python. If you experience any issues running the code or find any bugs, please let me know.

                                        Getting started

                                        One of the things that has stopped me from playing with deep learning in the past is the feeling that there is a bit of a steep learning curve around the tools and methods. A lot of the deep learning libraries out there don’t seem as mature as general machine learning libraries, such as scikit-learn. There are also many more parameters to play with when building deep neural networks than when using linear models or algorithms such as random forests. Further, to enable any kind of meaningful experimentation, using a GPU is essential.

                                        Fortunately, the tools and documentation have matured a lot in recent years. Motivated by Daniel Nouri’s excellent tutorial on detecting facial keypoints with convolutional neural nets, I decided to use the Lasagne package as my starting point. My plan was simple: Convert the MNIST example code to work on my dataset locally, setup an AWS machine with a GPU for full-scale experiments, and then play with various network architectures and techniques to improve accuracy and gain a deeper understanding of deep learning.

                                        Initial environment setup

                                        While Lasagne’s MNIST example code is pretty clear – especially once you get your head around the way Theano works – it doesn’t really lend itself to easy experimentation. I addressed this by refactoring the code in several iterations, until I got to the current state, where there’s a simple command-line interface that allows me to experiment with different datasets and architectures. This will probably change and become more complex as I start doing more sophisticated things.

                                        To enable rapid experimentation, I had to set up an AWS machine with a GPU (g2.2xlarge instance). I wrote some simple deployment code using Fabric, which allows me to setup a machine from scratch, install all the requirements, package the project, and copy it to the remote machine.

                                        Getting the code running on the CPU was trivial, but I hit several issues when running on the GPU. First, the vanilla Ubuntu 14.04 server I used didn’t come with CUDA installed. After trying and failing to get it working by following some tutorials, I ended up going down the easier path of using the AMI supplied by Caffe. This AMI also has the advantage of coming with Caffe installed (surprisingly), which I may end up using at some point.

                                        The second issue I encountered was that using the GPU to run Lasagne’s enhanced example code on my full dataset was impossible due to memory constraints. The problem was that the example assumes that the entire dataset can fit in the GPU’s memory (as discussed here and here). This took a while to resolve, even though the solution is conceptually simple – just copy the dataset to the GPU in chunks rather than attempt to copy it all in one go. Resolving this issue was a good way of getting a better understanding of what the code does, since I ended up rewriting most of the original example code.

                                        Next steps

                                        So far, I left the network architecture from the original example mostly untouched, as I was busy collecting the dataset, getting the environment set up, and resolving various issues. One thing I did notice was that the example’s architecture diverges on my dataset, so instead I tested my code using a basic multi-layer perceptron architecture with a single hidden layer. This performs about as well as a random classifier on my dataset, but at least it converges. I also tested the modified code on the MNIST dataset and the results are decent, so now it is time to move forward and actually do some modelling, starting with convolutional neural nets.

                                        The high level plan is to iteratively read tutorials/papers/books, implement ideas, play with parameters, and visualise parts of the network until I’m satisfied with the results. The main goal remains to learn as much as possible and get a good intuition of how things work. I’ll write more about my experiences in subsequent posts. Stay tuned!

                                        Update: The second post in the series is now available.

                                        Subscribe
                                          -
                                          Alternatively, subscribe to RSS feed.

                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/07/06/learning-about-deep-learning-through-album-cover-classification/index.html b/2015/07/06/learning-about-deep-learning-through-album-cover-classification/index.html index 5327ab5b6..967ef5361 100644 --- a/2015/07/06/learning-about-deep-learning-through-album-cover-classification/index.html +++ b/2015/07/06/learning-about-deep-learning-through-album-cover-classification/index.html @@ -22,7 +22,7 @@ soul" loading=lazy>

                                          The Kool-Aid Album by Mr. Merge
                                          soul

                                          dishrag, paper towel, honeycomb, envelope, chain mailsymbol, nobody, sign, illustration, color, flag, text, stripes, business, character

                                          Training from scratch

                                          My initial experiments were with various convnet architectures, where I manually varied the filter sizes and number of layers to have a reasonable number of parameters and ensure that the model is trainable on a GPU with 4GB of memory. As mentioned, this approach yielded unimpressive results. Following the relative success of the fine-tuned CaffeNet baseline, I decided to run more rigorous experiments on variants of AlexNet (which is very similar to CaffeNet).

                                          Given the large number of hyperparameters that need to be set when training deep convnets, I realised that setting values manually or via grid search is unlikely to yield the best results. To address this, I used hyperopt to search for the best configuration of values. The hyperparameters that were included in the search were the learning method (Nesterov momentum versus Adam with their respective parameters), the learning rate, whether crops are mirrored or not, the number of crops to use (1 or 5), dropout probabilities, the number of hidden units in the fully-connected layers, and the number of filters in each convolutional layer.

                                          Each configuration suggested by hyperopt was trained for 10 epochs, and the promising setups were trained until results stopped improving. The results of the search were rather disappointing, with the best accuracy being 17.19%. However, I learned a lot by finding hyperparameters in this manner – in the past I’ve only used a combination of manual settings with grid search.

                                          There are many possible reasons for why the results are so poor. It could be that there’s just too little data to train a good classifier, which is supported by the inability to beat the fine-tuned results. This is in line with the results obtained by Zeiler and Fergus (2013), who found that convnets pretrained on ImageNet performed much better on the Caltech-101 and Caltech-256 datasets than the same networks trained from scratch. However, it could also be that I just didn’t run enough experiments – I definitely feel like I haven’t explored everything as well as I’d like. In addition, I’m still building my intuition for what works and why. I should work more on visualising the way the network learns to uncover more hidden gotchas in addition to those I’ve already found. Finally, it could be that it’s just too hard to distinguish between covers from the genres I chose for the study.

                                          Ideas for future work

                                          There are many avenues for improving on the work I’ve done so far. The code could definitely be made more robust and better tested, optimised and parallelised. It would be worth investing more in hyperparameter and architecture search, including incorporation of ideas from non-vanilla convnets (e.g., GoogLeNet). This search should be guided by visualisation and a deeper understanding of the trained networks, which may also come from analysing class-level accuracy (certain genres seem to be easier to distinguish than others). In addition, more sophisticated preprocessing may yield improved results.

                                          If the goal were to get the best possible performance on my dataset, I’d invest in establishing the human performance baseline on the dataset by running some tests with Mechanical Turk. My guess is that humans would perform better than the algorithms tested so far due to access to external knowledge. Therefore, incorporating external knowledge in the form of manual features or additional data sources may yield the most substantial performance boosts. For example, text on an album cover may contain important clues about its genre, and models pretrained on style datasets may be more suitable than ImageNet models. In addition, it may be beneficial to use a model to detect multiple elements in images where the universe is not restricted to ImageNet classes. This approach was taken by Alexandre Passant, who used Clarifai’s API to tag and classify doom metal and K-pop album covers. Finally, using several different models in an ensemble is likely to help squeeze a bit more accuracy out of the dataset.

                                          Another direction that may be worth exploring is using image data for recommendation work. The reason I chose to work on this problem was my exposure to album covers through my work on Bandcamp Recommender – a music recommendation system. It is well-known that visual elements influence the way users interact with recommender systems. This is especially true in Bandcamp Recommender’s case, as users see the album covers before they choose to play them. This leads me to conjecture that considering features that describe the album covers when generating recommendations would increase user interaction with the system. However, it’s hard to tell whether it’d increase the overall relevance of the results. You can’t judge an album by its cover. Or can you…?

                                          Conclusion

                                          While I’ve learned a lot from working on this project, there’s still much more to discover. It was especially great to learn some generally-applicable lessons about hyperparameter optimisation and improvements to vanilla gradient descent. Despite the many potential ways of improving performance on my dataset, my next steps in the field would probably include working on problems for which obtaining a good solution is feasible and useful. For example, I have some ideas for applications to marine creature identification.

                                          Feedback and suggestions are always welcome. Please feel free to contact me privately or via the comments section.

                                          Acknowledgement: Thanks to Brian Basham and Diogo Moitinho de Almeida for useful tips and discussions.

                                          Subscribe
                                            -
                                            Alternatively, subscribe to RSS feed.

                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/07/31/goodbye-parse-com/index.html b/2015/07/31/goodbye-parse-com/index.html index a62001d59..eb0377c31 100644 --- a/2015/07/31/goodbye-parse-com/index.html +++ b/2015/07/31/goodbye-parse-com/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2015/07/31/goodbye-parse-com/bcrecommender-latency-digital-ocean.png 757w," src=https://yanirseroussi.com/2015/07/31/goodbye-parse-com/bcrecommender-latency-digital-ocean.png alt="BCRecommender latency with DigitalOcean" loading=lazy>

                                            Note that this graph is for a simple GET request of the homepage without fetching any of the embedded static assets or running client-side rendering. Handling the request simply populates a Jade template without touching the database. It really shouldn’t take too long unless the server is under very heavy load. And even then, Parse is supposed to handle such loads gracefully – not needing to worry about this kind of stuff is the key reason for using a backend-as-a-service!

                                            Final thoughts

                                            I really like the idea behind Parse, as setting up and running a web backend is not a trivial task. They do provide some good tooling, and I was happy to work around the minor issues and restrictions that come with working in a sandboxed environment. However, the lack of reliability is a huge disadvantage, even at the attractive price point of $0. Further, there’s no indication that paying for the service would increase reliability, as the free tier includes up to 30 requests / second and it can barely handle a single request. Maybe I’ll get back to Parse one day, but for now I’m much happier with the increased power and responsibility of managing my own servers.

                                            Update (30 January, 2016): Facebook has announced it will be shutting Parse down, which is a shame. It could have been a great service if they had just focused more on reliability. You just couldn’t run serious apps on Parse, which probably meant that not many apps were upgraded to the paid tiers. It’s very disappointing that Facebook didn’t help Parse realise its potential, but this isn’t the first time a big company takes over a small product and shuts it down. It’s just the way of the world.

                                            Subscribe
                                              -
                                              Alternatively, subscribe to RSS feed.

                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                              civax
                                              2015-08-14 11:06:12

                                              I think it’s all about what you expect. We used parse for prototypes and it’s been working great for us so far. I actually think we still have one of the prototypes running over it which we haven’t touched in almost year (a mobile/web strategy game now only available on FB - https://apps.facebook.com/foresttribes/). It’s been great since it also saved us the need to develop a backend admin tool to manage/balance the game or add additional content.

                                              Over all I never had a real live public product running on parse to comment on the experience, but for prototypes I’m perfectly happy with the service.

                                              Yanir Seroussi
                                              2015-08-15 08:16:14

                                              Agreed, it’s perfectly fine for prototypes, but a bit too unreliable for public-facing live products. If Parse were more robust, it’d be perfect for many use cases.
                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2015/08/24/you-dont-need-a-data-scientist-yet/index.html b/2015/08/24/you-dont-need-a-data-scientist-yet/index.html index 9f1a29e0f..7f37da185 100644 --- a/2015/08/24/you-dont-need-a-data-scientist-yet/index.html +++ b/2015/08/24/you-dont-need-a-data-scientist-yet/index.html @@ -2,7 +2,7 @@

                                              You don’t need a data scientist (yet)

                                              August 24, 2015

                                              The hype around big data has caused many organisations to hire data scientists without giving much thought to what these data scientists are going to do and whether they’re actually needed. This is a source of frustration for all parties involved. This post discusses some questions you should ask yourself before deciding to hire your first data scientist.

                                              Q1: Do you know what data scientists do?

                                              Somewhat surprisingly, there are quite a few companies that hire data scientists without having a clear idea of what data scientists actually do. People seem to have a fear of missing out on the big data hype, and think of hiring data scientists as the solution. A common misconception is that a data scientist’s role includes telling you what to do with your data. While this may sometimes happen in practice, the ideal scenario is where the business has problems that can be solved using data science (more on this under Q3 below). If you don’t know what your data scientist is going to do, you probably don’t need one.

                                              So what do data scientists do? When you think about it, adding the word “data” to “science” is a bit redundant, as all science is based on data. Following from this, anyone who does any kind of data analysis is a data scientist. While it may be true, this broad definition is not very helpful. As discussed in a previous post, it’s more useful to define data scientists as individuals who combine expertise in statistics and machine learning with strong software engineering skills.

                                              Q2: Do you have enough data available?

                                              It’s not uncommon to see products that suffer from over-engineering and premature investment in advanced analytics capabilities. In the early stages, it’s important to focus on creating a minimum viable product and getting it to market quickly. Data science starts to shine once the product is generating enough data, as most of the power of advanced analytics is in optimising and automating existing processes.

                                              Not having a data scientist in the early stages doesn’t mean the data is being ignored – it just means that it doesn’t require the attention of a full-time data scientist. If your product is at an early stage and you are still concerned, you’re better off hiring a data science consultant for a few days to help lay out the long-term vision for data-driven capabilities. This would be cheaper and less time-consuming than hiring a full-timer. The exception to this rule is when the product itself is built around advanced analytics (e.g., AlchemyAPI or Enlitic). Building such products without data scientists is far from ideal, or just impossible.

                                              Even if your product is mature and generating a lot of data, it doesn’t mean it’s ready for data science. Advanced analytics capabilities are at the top of data’s hierarchy of needs: If your product is buggy, or if your data is scattered everywhere and your platform lacks centralised reporting, you need to first invest in fixing your data plumbing. This is the job of data engineers. Getting data scientists involved when the data is hardly available due to infrastructure issues is likely to lead to frustration. In addition, setting up centralised reporting and dashboarding is likely to give you ideas for problems that data scientists can solve.

                                              Q3: Do you have a specific problem to solve?

                                              If the problem you’re trying to solve is “everyone is doing smart things with data, we should be doing stuff with data too”, you don’t have a specific problem that can be solved by bringing a data scientist on board. Defining the problem often ends up occupying a lot of the data scientist’s time, so you are likely to obtain better results if have more than just a vague idea around “doing something with data, because Hadoop”. Ideally you want to optimise an existing process that is currently being solved with heuristics, make an existing model better, implement a new data-driven feature, or something along these lines. Common examples include reducing churn, increasing conversions, and replacing manual processes with automated data-driven systems. Again, getting advice from experienced data scientists before committing to hiring one may be your best first step.

                                              Q4: Can you get away with heuristics, intuition, and/or manual processes?

                                              Some data scientists would passionately claim that you must deploy only models that are theoretically justified and well-tested. However, in many cases you can get away with using simple heuristics, intuition, and/or manual processes. These can be orders of magnitude cheaper than building sophisticated predictive models and the infrastructure to support them. For many businesses, there are more pressing needs than doing everything in a theoretically sound way. Despite what many technical people like to think, customers don’t tend to care how things are implemented, as long as their needs are fulfilled.

                                              For example, I spent some time with a client whose product includes a semi-manual part where structured data is extracted from documents. Their process included sending some of the documents to a trained team in the Philippines for manual analysis. The client was interested in replacing that manual work with a machine learning algorithm. As is often the case with machine learning, it was unknown whether the resultant model would be accurate enough to completely replace the manual workers. This generally depends on data quality and the feasibility of solving the problem. Assessing the feasibility would have taken some time and money, so the client decided to park the idea and focus on other areas of their business.

                                              Every business has resource constraints. Situations where the best investment you can make is hiring a full-time data scientist are rarer than what the hype may make you think. It’s often the case that functions that would be the responsibility of a data scientist are adequately performed by existing employees, such as software engineers, business/data analysts, and marketers.

                                              Q5: Are you committed to being data-driven?

                                              I have seen more than one case where data scientists are hired only to be blocked or ignored. This is more prevalent in the corporate world, where managers are often incentivised to prioritise doing things that look good over things that make financial sense. But even if recruitment is done with the best intentions, progress may be blocked by employees who feel threatened because they would be replaced by automated data-driven algorithms. Successful data science projects require support from senior leadership, as discussed by Greta Roberts, Radim Řehůřek, Alec Smith, and many others. Without such support and a strong commitment to making data-driven decisions, everyone is just wasting their time.

                                              Closing thoughts

                                              While data science is currently over-hyped, many organisations still have much to gain from hiring data scientists. I hope that this post has helped you decide whether you need a data scientist right now. If you’re unsure, please don’t hesitate to contact me. And to any data scientists reading this: Be very wary of potential employers who do not have good answers to the above questions. At this point in time you can afford to be picky, at least until the hype is over.

                                              Subscribe
                                                -
                                                Alternatively, subscribe to RSS feed.

                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                Eric Colson
                                                2015-08-28 16:10:40

                                                I enjoyed the post - though I offer some contrary points to consider:

                                                I have learned that if it is clear that you will need a data scientists (by someone who knows what they do), then you should get them as soon as possible. Don’t wait. Data Scientists work best when they have full context for the problem they are here to solve. Getting them in early allows them to help frame the problem. This framing is critical. If the framing is off, it takes a very long time (sometimes never) to get it back on track. A late-to-the-game data scientists can be too influenced by the the existing framing they are given. They tend to think within that box, when in reality, the box was never the right way to approach the problem. Even if they do see outside of it, it can be very difficult to convince the original framers that there is a better way to do things (people can get quite attached to their vision).

                                                It also can be wise to NOT WAIT till there is data to analyze. Too often, data is an afterthought. Its important for the data scientist to get in early on the initiative so he or she can help define the needed instrumentation and data acquisition strategy. They can even guide the needs of the data warehouse and other repositories where the newly captured data will reside.

                                                Further, it is often the case that it is the data scientist that identifies the specific problem to solve. At my company, I estimate that over half of the ideas for new data products, features, and services come from the data science team – not the business. This is intuitive as the data scientists are the folks that are most intimate with the data and are least constrained by what is possible to do with data. Give them business context and they will come up with problems/solutions that no one has thought of.

                                                Finally, I find heuristics to be dangerous. At best they are suboptimal, and more often than not, they are just plain wrong (those with extensive A/B testing experience can attest to the fact that our intuition fails us again and again). Undoing a bad heuristics can be very painful - in the technical work, the coordinate work, and in the resetting of expectations. Its hard to get people to not walk on a paved path … even if that path is the long way or a dead-end.

                                                I totally agree with “Q5: Are you committed to being data-driven?”. This comes down to business model and culture. Is your business model one where data science can be the source of strategic differentiation? Is your culture able to support empiricism? The answer to both of these has to be ‘yes’ in order to commit to being data-driven.

                                                Yanir Seroussi
                                                2015-08-29 02:43:41

                                                Thank you for your thoughtful comments, Eric!

                                                I generally agree that it can be beneficial to involve data scientists early on and to avoid thoughtless heuristics, but that it all depends on having a supportive data-driven environment and on resource constraints. As mentioned under Q2, getting advice from a data scientist in the early stages of the product is worthwhile, so it may be smart to pay for a few days of consulting, but not necessarily a good idea to hire a full-timer. A lot of it depends on the general product vision.

                                                Another note regarding heuristics and intuition: While some may be dangerous, you can view many modelling decisions as heuristics. For example, when building a predictive model, you have to make some intuition-driven choices around features (no model uses all the knowledge in the world), learning algorithms and their hyperparameters. You just can’t test everything, so there’s a need for compromises if you aim to ever deliver anything.

                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/10/02/the-wonderful-world-of-recommender-systems/index.html b/2015/10/02/the-wonderful-world-of-recommender-systems/index.html index 7e7b87782..5b150e39e 100644 --- a/2015/10/02/the-wonderful-world-of-recommender-systems/index.html +++ b/2015/10/02/the-wonderful-world-of-recommender-systems/index.html @@ -18,7 +18,7 @@ 100vw" srcset="https://yanirseroussi.com/2015/10/02/the-wonderful-world-of-recommender-systems/giveable-logo.png 250w," src=https://yanirseroussi.com/2015/10/02/the-wonderful-world-of-recommender-systems/giveable-logo.png alt="Giveable logo" loading=lazy>

                                                At the time, there wasn’t much published research on gift recommendation, and there was more or less nothing about the specific problem of recommending gifts for Facebook friends using liked pages. Here are some of the ways this problem differs from classic recommendation scenarios.

                                                • Need to consider giver and receiver. Unlike traditional scenarios, the recommended items aren’t consumed by the user to whom they’re shown. In practice, this meant that we had to ensure the items are giftable, and take into account the relationship between the giver and the receiver. For example, the type of gift your mum may give you is different from gifts your partner may give you.
                                                • Likes are historical, sparse, and often nonsensical. This is best illustrated by an example: What does liking a page such as Tony Abbott – Worst PM in Australian History tell us about gifts the user may like? Tony Abbott is no longer prime minister (thankfully), so it’s historical, and while this page is quite popular, there are many other pages out there that are difficult to interpret and are liked by only a handful of people (this video is a good summary of why Tony is disliked, for those who are unfamiliar with Australian politics).
                                                • Likes are not for recommended items. As the above example shows, just because you like disliking Tony, it doesn’t exactly lead to useful gifts. Even with things that are more related to interests, such as authors and bands, the liked pages aren’t recommendable as gifts.
                                                • Likes are not always available offline. This was an important engineering consideration: We didn’t have much time to generate recommendations from the point where a new user gave us permission to view their likes and the likes of their friends. Ideally, recommendation generation would take less than a second from the time we got all the data from Facebook. This puts a strong constraint on the types of algorithms we could use.

                                                The key to effectively addressing the Giveable recommendation problem was doing as much processing offline as possible. Specifically:

                                                • Similar pages were inferred using Latent Dirichlet Allocation (which can be seen as a collaborative filtering technique). This made it possible to use information from pages that are not directly linked to giftable products, e.g., for the above Tony Abbott example, people who dislike him are likely to be left-leaning, which implies many other interests.
                                                • Facebook pages were matched to giftable products with heuristics + Mechanical Turk + machine learning. This took a few iterations of what was essentially partly-manual semi-supervised learning, where we obtained high-confidence matches through heuristics and manual tagging, and then used this to train a classifier that was used to classify uncertain matches. The results of classification on a hold-out set were then verified through manual tagging of subsamples.
                                                • We enriched the page and product data with structured information from the Freebase knowledge graph (which has since been deprecated). This allowed us to easily match giftable products to liked pages, e.g., books to authors.

                                                The online part included taking a receiver’s liked pages, inferring likes for similar pages, and matching all these pages to a ranked and diversified list of giftable product recommendations. These recommendations came with explanations, which were quite important in this case because the giver of a gift has to know why they’re giving it.

                                                The silver bullet myth
                                                Optimising a single measure or using a single algorithm is sufficient for generating a good recommendation list
                                                Reality
                                                Hybrids work best

                                                Netflix provides another example for how focusing on a single algorithm or measure of success is far from sufficient. In a recent blog post, they talk about how they use multiple algorithms to optimise the order of different recommendation lists and each list’s internal ranking, while considering device-specific UI constraints, relevance, engagement, diversity, business requirements, and more.

                                                An example from my experience comes from Giveable (which ended up evolving into Hynt), where a single list was generated by mixing the outputs of the following recommendation approaches: contextual, direct likes, inferred likes, content-based, social, collaborative filtering of products, previously viewed items, and popular interests/products. The weight of each algorithm in the mix was static – it was either set manually or through A/B testing, and then left as a hardcoded constant.

                                                This kind of static mix can get you very far, but there’s a better way that I haven’t gotten around to implementing before leaving to work on other things. This way is described in a series of posts on bandits for recommenders by Sergey Feldman of RichRelevance. The general idea is to train recommendation models offline using a small number of strategies/paradigms. Online, recommendations are served from strategies that maximise clickthrough and revenue, given a context of features that describe the user, merchant, and web page where the RichRelevance widget is embedded. Rather than setting static weights for the strategies, the bandit model continuously adjusts the weights, while balancing between exploring new strategy weights and exploiting strategies that have been known to work well in a specific context. This allows the overall recommendation engine to adjust to changes in reality and in the underlying data.

                                                The omnipresence myth
                                                Every personalised system is a recommender system
                                                Reality
                                                This one is kinda true, but not necessarily useful...

                                                The first conference I attended as a PhD student was the 18th International Conference on User Modeling, Adaptation and Personalization (UMAP), back in 2010. The field of recommender systems was getting increased attention, and Peter Brusilovsky, who has been working in the UMAP field for decades, argued that recommender systems are the new expert systems. This was partly because the hype was causing people to broaden the definition of the field to allow them to say that they’re working on recommender systems.

                                                I don’t think it’s incorrect that personalisation and recommender systems are different things. However, one problem that this may cause is making people think that common recommendation techniques would apply in scenarios where they’re unlikely to work. For example, web search can be seen as a recommender system for pages that gives a high weight to the user’s intent, as captured by the query. Hence, when personalising web search, it seems sensible to use collaborative filtering techniques. This was indeed my experience with the Yandex search personalisation competition: employing a matrix factorisation approach that was inspired by collaborative filtering turned out to be a waste of time compared to domain-specific methods.

                                                In conclusion, recommenders are about as murky as data science. Just like data science, the boundaries of recommender systems are hard to define and they are sometimes over-hyped. This hype may lead to people investing in a recommender system they don’t really need, just like the common issue of premature investment in data science. However, the hype is based on real value, which can definitely be delivered by recommender systems when they are used correctly.

                                                Subscribe
                                                  -
                                                  Alternatively, subscribe to RSS feed.

                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                  Zygmunt
                                                  2015-10-02 16:51:09

                                                  How did you arrive at the conclusion that accuracy doesn’t matter? The Netflix quote and chart don’t seem connected to me. The quote refers to the massive ensembling used to achieve the challenge score threshold. The chart seems to show that you can go a long way from the baseline by improving features and models.

                                                  I’d say accuracy, or more generally, score used for evaluation, doesn’t matter as long as it’s good enough. However, it’s not that easy to arrive at “good enough”. Consider Spotify. I find their daily recommendations abysmal. Discover Weekly is much better, but still has room to improve.

                                                  Yanir Seroussi
                                                  2015-10-02 20:36:57

                                                  I agree. I said that predictive accuracy has some importance, but it is not the only thing that matters. You’re right about it needing to be good enough, where the definition of good enough is domain-dependent.

                                                  Daniel Lopresti said it well years ago (he spoke about web search but it applies to recommendation scenarios where suggestions are browsable):

                                                  Browsing is a comfortable and powerful paradigm (the serendipity effect). Search results don't have to be very good. Recall? Not important (as long as you get at least some good hits). Precision? Not important (as long as at least some of the hits on the first page you return are good).

                                                  flower
                                                  2016-01-26 18:47:02

                                                  it does not clear to me why accuracy is not important in recommender and searching??

                                                  Yanir Seroussi
                                                  2016-01-26 19:21:55

                                                  It is important, but its importance tends to be exaggerated to the exclusion of all other metrics. As I said in the post, things like the way you present your results (UI/UX) and novelty/serendipity are also very important. In addition, the goal of the system is often to optimise a different goal from offline accuracy, such as revenue or engagement. In such cases it is best to focus on what you want to improve rather than offline accuracy.

                                                  By the way, I attended a talk by Ted Dunning a few months ago, where he said that one of the most important tweaks in real-life recommender system is adding random recommendations (essentially decreasing offline accuracy). This allows the system to learn from user feedback on a wider range of items, improving performance in the long run.

                                                  flower
                                                  2016-01-26 21:00:40

                                                  Thank you very much for your fast response. diff --git a/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/index.html b/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/index.html index d803111d7..81bd266b5 100644 --- a/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/index.html +++ b/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/index.html @@ -5,7 +5,7 @@ https://yanirseroussi.com/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/phd-comics-science-news-cycle.gif 600w," src=https://yanirseroussi.com/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/phd-comics-science-news-cycle.gif alt="PHD Comics: Science News Cycle" loading=lazy>

                                                  Selling your model with simple explanations

                                                  People like simple explanations for complex phenomena. If you work as a data scientist, or if you are planning to become/hire one, you’ve probably seen storytelling listed as one of the key skills that data scientists should have. Unlike “real” scientists that work in academia and have to explain their results mostly to peers who can handle technical complexities, data scientists in industry have to deal with non-technical stakeholders who want to understand how the models work. However, these stakeholders rarely have the time or patience to understand how things truly work. What they want is a simple hand-wavy explanation to make them feel as if they understand the matter – they want a story, not a technical report (an aside: don’t feel too smug, there is a lot of knowledge out there and in matters that fall outside of our main interests we are all non-technical stakeholders who get fed simple stories).

                                                  One of the simplest stories that most people can understand is the story of correlation. Going back to the running example of predicting health based on diet, it is well-known that excessive consumption of certain fats under certain conditions is correlated with an increase in likelihood of certain diseases. This is simplified in some stories to “consuming more fat increases your chance of disease”, which leads to the conclusion that consuming no fat at all decreases the chance of disease to zero. While this may sound ridiculous, it’s the sad reality. According to a recent survey, while the image of fat has improved over the past few years, 42% of Americans still try to limit or avoid all fats.

                                                  A slightly more involved story is that of linear models – looking at the effect of the most important factors, rather than presenting a single factor’s contribution. This storytelling technique is commonly used even with non-linear models, where the most important features are identified using various techniques. The problem is that people still tend to interpret this form of presentation as a simple linear relationship. Expanding on the previous example, this approach goes from a single-minded focus on fat to the need to consume less fat and sugar, but more calcium, protein and vitamin D. Unfortunately, even linear models with tens of variables are hard for people to use and follow. In the case of nutrition, few people really track the intake of all the nutrients covered by recommended daily intakes.

                                                  Few interesting relationships are linear

                                                  Complex phenomena tend to be explained by complex non-linear models. For example, it’s not enough to consume the “right” amount of calcium – you also need vitamin D to absorb it, but popping a few vitamin D pills isn’t going to work well if you don’t consume them with fat, though over-consumption of certain fats is likely to lead to health issues. This list of human-friendly rules can go on and on, but reality is much more complex. It is naive to think that it is possible to predict something as complex as human health with a simple linear model that is based on daily nutrient intake. That being said, some relationships do lend themselves to simple rules of thumb. For example, if you don’t have enough vitamin C, you’re very likely to get scurvy, and people who don’t consume enough vitamin B1 may contract beriberi. However, when it comes to cancers and other diseases that take years to develop, linear models are inadequate.

                                                  An accurate model to predict human health based on diet would be based on thousands to millions of variables, and would consider many non-linear relationships. It is fairly safe to assume that there is no magic bullet that simply explains how diet affects our health, and no superfood is going to save us from the complexity of our nutritional needs. It is likely that even if we had such a model, it would not be completely accurate. All models are wrong, but some models are useful. For example, the vitamin C versus scurvy model is very useful, but it is often wrong when it comes to predicting overall health. Predictions made by useful complex models can be very hard to reason about and explain, but it doesn’t mean we shouldn’t use them.

                                                  The ongoing quest for sellable complex models

                                                  All of the above should be pretty obvious to any modern data scientist. The culture of preferring complex models with high predictive accuracy to simplistic models with questionable predictive power is now prevalent (see Leo Breiman’s 2001 paper for a discussion of these two cultures of statistical modelling). This is illustrated by the focus of many Kaggle competitions on producing accurate models and the recent successes of deep learning for computer vision. Especially with deep learning for vision, no one expects a handful of variables (pixels) to be predictive, so traditional explanations of variable importance are useless. This does lead to a general suspicion of such models, as they are too complex for us to reason about or fully explain. However, it is very hard to argue with the empirical success of accurate modelling techniques.

                                                  Nonetheless, many data scientists still work in environments that require simple explanations. This may lead some data scientists to settle for simple models that are easier to sell. In my opinion, it is better to make up a simple explanation for an accurate complex model than settle for a simple model that doesn’t really work. That being said, some situations do call for simple or inflexible models due to a lack of data or the need to enforce strong prior assumptions. In Albert Einstein’s words, “it can scarcely be denied that the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience”. Make things as simple as possible, but not simpler, and always consider the interests of people who try to sell you simplistic (or unnecessarily complex) explanations.

                                                  Subscribe
                                                    -
                                                    Alternatively, subscribe to RSS feed.

                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/index.html b/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/index.html index daad58333..9329cf858 100644 --- a/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/index.html +++ b/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/index.html @@ -10,7 +10,7 @@ https://yanirseroussi.com/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/bcrecommender-search-amanda.png 1202w," src=https://yanirseroussi.com/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/bcrecommender-search-amanda_hu0232655e13350a3ced90992236bca155_32800_800x0_resize_box_3.png alt="bcrecommender search for amanda" loading=lazy>

                                                    I deployed full-text search earlier this week, and so far it’s looking pretty good. Elasticsearch seems to be coping well with having the same level of resources allocated as before, but it’s still too early to tell if this is sustainable over time. Most importantly, users are finally seeing results when they enter unstructured queries, which increases their engagement and retention. Woohoo!

                                                    Improving search performance based on user behaviour is expected to be an ongoing effort. Despite having many ideas, I resisted the temptation of endless offline tinkering and opted to release a working search page quickly. With Google Analytics now set up to track site search, the plan is keep identifying gaps and tweak the search settings continuously. This will take a while, as the number of daily users is currently 200-300, and they don’t all use site search.

                                                    Implementing more search features is another set of items on my to-do list that will be addressed over time. For example, it’d be great to have search auto-completion and a prettier result page. However, I have more ideas than time to implement them, and I’m not working on BCRecommender full-time. For now, I’m pretty happy with finally having the search function.

                                                    Elasticsearch versus MongoDB: Key findings

                                                    Comparisons between tools should always be taken with a grain of salt. General comparisons may not address features that are important for your specific use case, or may overemphasise aspects that you don’t care about. In addition, actively developed tools are moving targets. Since I started the transition to Elasticsearch, version 2.0 has been released, and MongoDB 3.2 is expected very soon. The following list is derived from my experience and may not apply to you. You have been warned!

                                                    With the disclaimer out of the way, here are some of the advantages of Elasticsearch over MongoDB:

                                                    • Better full-text search support (duh!).
                                                    • Enforceable schemas and type validation (note: some form of optional schema is expected in MongoDB 3.2).
                                                    • All fields are indexed by default, making it easy to explore unstructured data without worrying about adding indices.
                                                    • It appears that indexing is implemented in a more efficient way that doesn’t block the node. Slowness due to indexing operations seems to be a common issue with MongoDB, even with background index creation.
                                                    • It’s possible to query multiple indices and types (same as MongoDB databases & collections, respectively) in the same query. This is a huge advantage in my case as it makes it possible to efficiently search both fans and tralbums in a single query.
                                                    • Index aliases make it easy to change the indices without changing the application.
                                                    • Multi-get by IDs returns results in the order they were requested. This is not the case with MongoDB, where using $in doesn’t have any guarantees on the returned documents’ order. It’s easy to work around this issue, but it can be the source of subtle bugs. In my case, recommendations were unintentionally sorted in random order until I added an additional step to sort them correctly.
                                                    • Built-in support for random scoring (note: random sampling will finally be available in MondoDB 3.2 – the ticket for this has been open for 5 years).
                                                    • Built-in support for multiple types of analysis on the same field.

                                                    Some disadvantages of Elasticsearch in comparison to MongoDB are:

                                                    • All fields are indexed by default, making it easy to run into memory issues. Adjusting these default settings is strongly recommended if you know how you’re going to query the data.
                                                    • Documents are immutable, so every update requires deleting the original document and re-inserting it (in practice, it seems like this isn’t much of an issue).
                                                    • Sorting results by a field requires reading all the field’s values and sorting them in memory. The sorted results are cached, but this may cause issues if memory is too limited.

                                                    In conclusion, my experience with Elasticsearch has been mostly positive so far and I’m glad I’ve made the switch. I’m looking forward to taking further advantage of advanced search features to improve user experience on BCRecommender. New posts on the topic may be published in the future, so please subscribe to be notified when this happens. As always, I’m happy to receive feedback through the comments or privately.

                                                    Subscribe
                                                      -
                                                      Alternatively, subscribe to RSS feed.

                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2015/11/23/the-hardest-parts-of-data-science/index.html b/2015/11/23/the-hardest-parts-of-data-science/index.html index 613585c86..87d711844 100644 --- a/2015/11/23/the-hardest-parts-of-data-science/index.html +++ b/2015/11/23/the-hardest-parts-of-data-science/index.html @@ -9,7 +9,7 @@ https://yanirseroussi.com/2015/11/23/the-hardest-parts-of-data-science/dilbert-big-data.jpg 904w," src=https://yanirseroussi.com/2015/11/23/the-hardest-parts-of-data-science/dilbert-big-data_hu52b7900d669ddfe03bdd894fda67e436_330047_800x0_resize_q75_box.jpg alt="Dilbert big data" loading=lazy>

                                                      Well-defined problems are great, for the obvious reason that they can actually be addressed. Examples of such problems include:

                                                      • Build a model to predict the sales of a marketing campaign
                                                      • Create a system that runs campaigns that automatically adapt to customer feedback
                                                      • Identify key objects in images
                                                      • Improve click-through rates on search engine results, ads, or any other element
                                                      • Detect whale calls from underwater recordings to prevent collisions

                                                      Often, it can be hard to get to the stage where the problem is agreed on, because this requires dealing with people who only have a fuzzy idea of what can be done with data science. Dilbertian situations aside, these people often have real problems that they care about, so exploring the core issues with them is time well-spent.

                                                      Solution measurement is often harder than problem definition

                                                      Many problems that actually matter have solutions that are really hard to measure. For example, improving the well-being of the population (e.g., a company’s customers or a country’s citizens) is an overarching problem that arises in many situations. However, this problem gives rise to the hard question of how well-being can be measured and aggregated. The following paragraphs discuss issues that occur in solution measurement, often making it the hardest part of data science.

                                                      Ideally, we would always be able to run randomised controlled trials to measure treatment effects. However, the reality is that experimental data is often censored, there many constraints on running experiments (ethics, practicality, budget, etc.), and confounding factors may make it impossible to identify the true causal impact of interventions. These issues seriously influence many aspects of our lives. I’ve written a post on how these issues manifest themselves in research on the connection between nutrition and our health. Here, I’ll discuss another major example: the health effects of smoking and anthropogenic climate change.

                                                      While smoking and anthropogenic climate change may seem unrelated, they actually have a lot in common. In both cases it is hard (or impossible) to perform experiments to determine causality, and in both cases this fact has been used to mislead the public by parties with commercial and ideological interests. In the case of smoking, due to ethical reasons, one can’t perform an experiment where a random control group is forced not to smoke, while a treatment group is forced to smoke. Further, since it can take many years for smoking-caused diseases to develop, it’d take a long time to obtain the results of such an experiment. Tobacco companies have exploited this fact for years, claiming that there may be some genetic factor that causes both smoking and a higher susceptibility to smoking-related diseases. Fortunately, we live in a world where these claims have been widely discredited, and it is now clear to most people that smoking is harmful. However, similar doubt-casting techniques are used by polluters and their supporters in the debate on anthropogenic climate change. While no serious climate scientist doubts the fact that human activities are causing climate change, this can’t be proved through experimentation on another Earth. In both cases, the answers should be clear when looking at the evidence and the mechanisms at play without an ideological bias. It doesn’t take a scientist to figure out that pumping your lungs full of smoke on a regular basis is likely to be harmful, as is pumping the atmosphere full of greenhouse gases that have been sequestered for millions of years. However, as said by Upton Sinclair, “it is difficult to get a man to understand something, when his salary depends upon his not understanding it.”

                                                      Assuming that we have addressed the issues raised so far, there is the matter of choosing a measure or metric of success. How do we know that our solution works well? A common approach is to choose a single metric to focus on, such as increasing conversion rates. However, all metrics have their flaws, and there are quite a few problems with metric selection and its maintenance over time.

                                                      First, focusing on a single metric can be harmful, because no metric is perfect. A classic example of this issue is the focus on growing the economy, as measured by gross domestic product (GDP). The article What is up with the GDP? by Frank Shostak summarises some of the problems with GDP:

                                                      The GDP framework cannot tell us whether final goods and services that were produced during a particular period of time are a reflection of real wealth expansion, or a reflection of capital consumption.

                                                      For instance, if a government embarks on the building of a pyramid, which adds absolutely nothing to the well-being of individuals, the GDP framework will regard this as economic growth. In reality, however, the building of the pyramid will divert real funding from wealth-generating activities, thereby stifling the production of wealth.

                                                      […]

                                                      The whole idea of GDP gives the impression that there is such a thing as the national output. In the real world, however, wealth is produced by someone and belongs to somebody. In other words, goods and services are not produced in totality and supervised by one supreme leader. This in turn means that the entire concept of GDP is devoid of any basis in reality. It is an empty concept.

                                                      Shostak’s criticism comes from a right-winged viewpoint – his argument is that the GDP is used as an excuse for unnecessary government intervention with the market. However, the focus on GDP growth is also heavily-criticised by the left due to the fact that it doesn’t consider environmental effects and inequalities in the distribution of wealth. It is a bit odd that GDP growth is still considered a worthwhile goal by many people, given that it can easily be skewed by a few powerful individuals who choose to build unnecessary pyramids (though perhaps this is the real reason why the GDP persists – wealthy individuals have an interest in keeping it this way).

                                                      Even if we decide to use multiple metrics to evaluate our solution, our troubles aren’t over yet. Using multiple metrics often means that there are trade-offs between the different metrics. For example, with the precision and recall measures that are commonly used to evaluate the performance of search engines, it is rare to be able to increase both precision and recall at the same time. Precision is the percentage of relevant items out of those that have been returned, while recall is the percentage of relevant items that have been returned out of the overall number of relevant items. Hence, it is easy to artificially increase recall to 100% by always returning all the items in the database, but this would mean settling for near-zero precision. Similarly, one can increase precision by always returning a single item that the algorithm is very confident about, but this means that recall would suffer. Ultimately, the best balance between precision and recall depends on the application.

                                                      Another issue with choosing metrics is the impossibility of reliably evaluating our choices. This is summarised well by Scott Berkun in his book The Year Without Pants:

                                                      All metrics create temptations. Even with great intentions and smart minds, data runs you faster and faster into a stupid self-destructive circle. Data can’t decide things for you. It can help you see things more clearly if captured carefully, but that’s not the same as deciding. Just as there is an advice paradox, there is a data paradox: no matter how much data you have, you still depend on your intuition for deciding how to interpret and then apply the data.

                                                      Put another way, there is no good KPI for measuring KPIs. There are no good metrics for evaluating metrics (or for evaluating metrics for evaluating metrics for evaluating metrics, and on it goes).

                                                      OK, so we’ve picked some flawed measures that we can’t really evaluate, and we’ve accepted the imperfections of the evaluation process. Are we done yet? No. There’s still the small matter of Goodhart’s Law, which states that “when a measure becomes a target, it ceases to be a good measure.” This is often the case because people will tend to manipulate results and game the system (not necessarily maliciously) in order to hit measured goals. However, even without manipulation and gaming, we often deal with moving targets. Just because the measure we’ve chosen is suitable today, it doesn’t mean it will still be relevant in a few months or years because reality changes. For example, in the 1990s, the number of page views was a good measure of interaction with websites, but nowadays it is a pretty weak measure because many websites are single-page applications. Reality changes and so should our problems, solutions, measures, and goals.

                                                      Embracing ambiguity and uncertainty

                                                      Personally, I find the complexities of measurement and problem definition quite interesting. However, many people aren’t that interested in this stuff – they just want working solutions and simple stories. As demonstrated by the examples throughout this article, over-simplification of complicated matters is a pervasive issue that goes beyond what’s commonly considered “data science”. This is why storytelling is seen as a key skill that data scientists should possess. I believe it’s also important to maintain one’s integrity and not just make up stories that people would buy, but it’d be naive to assume that this never happens. Either way, good data scientists embrace uncertainty and ambiguity, but can still tell a simple story if needed.

                                                      Note: The ideas in this post were first presented at The Sydney Data Science Breakfast Meetup Group. The slides for that talk are available here.

                                                      Subscribe
                                                        -
                                                        Alternatively, subscribe to RSS feed.

                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                        RG
                                                        2015-11-24 02:37:44

                                                        Excellent excellent common sense article which seems to be very uncommon nowadays in a hype-filled world. Thanks for reminding us that accurate problem description should trump everything else!

                                                        Dalila
                                                        2015-11-30 14:50:28

                                                        Thank you for a great article. Yes, well defined problems and well defined performance evaluation are keys to designing any data driven model.

                                                        I also found that sometimes we have the question we want to pursue, but getting to an answer is not straight forward. For instance, I’m trying to find affinity between food ingredients using only data analytics. One may think that this problem is trivial. In fact, to find this, one has to totally rethink how to represent data (having ingredients in a table or a dataset produced nothing.) Yes finding affinity between 2 ingredients is trivial, but when the number grows up, one has to change the setting. In my case, I had to think of ingredients as part of a complete network, where the network is a recipe. It is then and only then, that I was able to find affinity between many ingredients.

                                                        andrew
                                                        2015-12-04 00:29:33

                                                        Yes, many good points here, thanks for this. There is even another difficulty apart from problem definition and solution measurement: the semantics of the data itself. Are the definitions real (referring to other concepts) or nominal (“a cheeseburger is a burger with cheese”)? Scope and context can easily be lost, and can only be put back by a human being taking a decision, no amount of empirical modelling can re-discover this. Also precision and accuracy of the data may be unknown and/or insufficient to solve the problem posed. If you have detected these issues, sometimes you can re-formulate the problem,but typically its not clear from the column headings alone (if you even have these). Even worse, the definitions may be incoherent or nonsensical: eg in classical econometric modelling the definition of a rational agent entails that the agent have knowledge of the future!

                                                        Yanir Seroussi
                                                        2015-12-06 05:44:13

                                                        Thanks Andrew! I agree that often what you can do is very limited by the data. I’ve also encountered cases where I had to infer meaning from cryptic column names. In many cases the small arbitrary decisions that we make along the way can have a major influence on the final results!

                                                        Arthur
                                                        2015-12-08 00:28:04

                                                        Is this article a Poe? The amount of muddled priors throughout it is disturbing. The word “sophistry” keeps leaping to mind. E.g.:

                                                        > For instance, if a government embarks on the building of a pyramid […] s/“a government”/Paris/g diff --git a/2015/12/08/this-holiday-season-give-me-real-insights/index.html b/2015/12/08/this-holiday-season-give-me-real-insights/index.html index 1f755fa50..a78ef0cb9 100644 --- a/2015/12/08/this-holiday-season-give-me-real-insights/index.html +++ b/2015/12/08/this-holiday-season-give-me-real-insights/index.html @@ -14,7 +14,7 @@ https://yanirseroussi.com/2015/12/08/this-holiday-season-give-me-real-insights/linkedin-profile-views.png 983w," src=https://yanirseroussi.com/2015/12/08/this-holiday-season-give-me-real-insights/linkedin-profile-views_hu9bee2f11dc6b74171edf6cb72c9c5a93_38858_800x0_resize_box_3.png alt="LinkedIn profile views" loading=lazy>

                                                        What would real LinkedIn insights look like? First, I think that the focus on profile views is somewhat misguided. It’s not that hard to artificially generate profile views – simply view other people’s profiles. There is no intrinsic value in someone having viewed your profile – the value comes from a connection that leads to an interesting offer or conversation. Second, LinkedIn is about professional networking that is based on real-world activity. As such, it only forms a small part of the world of professional networking by allowing people to have an online presence that makes them contactable by people they don’t already know. When it comes to insights, it’d be useful to know the true causal factors that lead to interesting connections – much more useful than suggestions such as add software development as a skill on your profile to get up to 3% more profile views.

                                                        Summary: Real insights are about the why

                                                        There are many other examples of pseudo-insights out there. The reason is probably that the field of analytics is becoming increasingly commoditised, and it is easier to rebrand an analytics dashboard as an insights dashboard than to provide real insights. Providing real insights requires moving up the DIKW pyramid from data and information to knowledge and wisdom – from describing the past to learning general lessons that allow you to influence the future. Providing real insights can be very hard, as it often requires inferring the causes of events – the why that comes after the what and how. More on this later – I have just started reading Samantha Kleinberg’s Why: A Guide to Finding and Using Causes and will report (hopefully real) insights on causality in future posts.

                                                        Subscribe
                                                          -
                                                          Alternatively, subscribe to RSS feed.

                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                          Greg Ichneumon Brown
                                                          2015-12-21 22:52:05

                                                          Nice post. Mostly agree. Automated insights are hard to automate though, but we (the WordPress.com Data Team) are working on it.

                                                          Some of the things we’ve found that have the biggest impact on successful blogging are:

                                                          • Turning on publicize so your posts are pushed out to various social media channels
                                                          • Regularly publishing. Doesn’t have to be daily, but does need to be regular. We still don’t understand how the periodicity plays into this.
                                                          • Images in posts are correlated with more traffic.

                                                          There’s still a lot to learn here. Interested in helping? https://automattic.com/work-with-us/data-wrangler/ :)

                                                          Yanir Seroussi
                                                          2015-12-27 19:48:26

                                                          Thanks Greg! All those factors make sense. Personally, I prefer sharing posts manually to turning on Publicize, but I suppose it has the same effect. My guess is that one of the reasons why images are important is that having at least one image makes posts stick out when shared on social media.

                                                          By the way, I did apply for the data wrangler position a couple of months ago but never heard back. It’s probably too late now, as I have a different position (and a few other options) lined up when I get home from vacation next month :)

                                                          Greg Ichneumon Brown
                                                          2015-12-28 04:26:19

                                                          Hey Yanir, that’s embarrassing. :)

                                                          Sorry we haven’t gotten back to you yet. I do see you in our queue. Its been a busy two months so we’re a bit backed up, but getting back on track in the next week or two. Certainly understand if that doesn’t fit into your own timeline. Sorry if that ends up being the case.

                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2016/01/24/the-joys-of-offline-data-collection/index.html b/2016/01/24/the-joys-of-offline-data-collection/index.html index 0d4bf57ad..57f2be0ae 100644 --- a/2016/01/24/the-joys-of-offline-data-collection/index.html +++ b/2016/01/24/the-joys-of-offline-data-collection/index.html @@ -18,7 +18,7 @@ https://yanirseroussi.com/2016/01/24/the-joys-of-offline-data-collection/rls-survey-counts.png 777w," src=https://yanirseroussi.com/2016/01/24/the-joys-of-offline-data-collection/rls-survey-counts.png alt="RLS surveys by Australian financial year (July-June)" loading=lazy>

                                                          RLS surveys by Australian financial year (July-June). Source: RLS Foundation Annual Report 2015

                                                          Each RLS survey requires several hours of work. In addition to performing the survey itself, a lot of work goes into entering the data and verifying its quality. Getting to the survey sites is not always a trivial task, especially for remote sites such as some of those we dived on my recent trip. Spending a month diving the Great Barrier Reef is a good way of appreciating its greatness. As the map shows, the surveys we did covered only the top part of the reef’s 2300 kilometres, and we only sampled a few sites within that part. The Great Barrier Reef is very vast, and it is hard to convey its vastness with just words or a map. You have to be there to understand – it is quite humbling.

                                                          In summary, the RLS experience has given me a new appreciation for small data in the offline world. Offline data collection is often expensive and labour-intensive – you need to work hard to produce a few high-quality data points. But the size of your data doesn’t matter (though having more quality data is always good). What really matters is what you do with the data – and the RLS team and their collaborators have been doing quite a lot. The RLS experience also illustrates the importance of domain expertise: I’ve looked at the RLS datasets, but I have no idea what questions are worth asking and answering using those datasets. The RLS project is yet another example of how in science collecting data is time-consuming, and coming up with appropriate research questions is hard. It is a lot of fun, though.

                                                          Subscribe
                                                            -
                                                            Alternatively, subscribe to RSS feed.

                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/index.html b/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/index.html index 7802746e9..94ba050e2 100644 --- a/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/index.html +++ b/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/us-science-spending-versus-suicides_hucb19a666efd495d868358d2e56c5c43f_82139_1500x0_resize_box_3.png 1500w," src=https://yanirseroussi.com/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/us-science-spending-versus-suicides_hucb19a666efd495d868358d2e56c5c43f_82139_800x0_resize_box_3.png alt="US science spending versus suicides" loading=lazy>

                                                            Source: Spurious Correlations by Tyler Vigen

                                                            Causal analysis aims to identify factors that are independent of spurious correlations, allowing stakeholders to make well-informed decisions. It is all about getting to the top of the DIKW (data-information-knowledge-wisdom) pyramid by understanding why things happen and what we can do to change the world. However, finding true causes can be very hard, especially in cases where you can’t perform experiments. Judea Pearl explains it well:

                                                            We know, from first principles, that any causal conclusion drawn from observational studies must rest on untested causal assumptions. Cartwright (1989) named this principle ‘no causes in, no causes out,’ which follows formally from the theory of equivalent models (Verma and Pearl, 1990); for any model yielding a conclusion C, one can construct a statistically equivalent model that refutes C and fits the data equally well.

                                                            What this means in practice is that you can’t, for example, conclusively prove that smoking causes cancer without making some reasonable assumptions about the mechanisms at play. For ethical reasons, we can’t perform a randomly controlled trial where a test group is forced to smoke for years while a control group is forced not to smoke. Therefore, our conclusions about the causal link between smoking and cancer are drawn from observational studies and an understanding of the mechanisms by which various cancers develop (e.g., the effect of cigarette smoke on individual cells can be studied without forcing people to smoke). Cancer Tobacco companies have exploited this fact for years, making the claim that the probability of both cancer and smoking is raised by some mysterious genetic factors. Fossil fuel and food companies use similar arguments to sell their products and block attempts to regulate their industries (as discussed in previous posts on the hardest parts of data science and nutritionism). Fighting against such arguments is an uphill battle, as it is easy to sow doubt with a few simplistic catchphrases, while proving and communicating causality to laypeople is much harder (or impossible when it comes to deeply-held irrational beliefs).

                                                            My causality journey is just beginning

                                                            My interest in formal causal analysis was seeded a couple of years ago, with a reading group that was dedicated to Judea Pearl’s work. We didn’t get very far, as I was a bit disappointed with what causal calculus can and cannot do. This may have been because I didn’t come in with the right expectations – I expected a black box that automatically finds causes. Recently reading Samantha Kleinberg’s excellent book Why: A Guide to Finding and Using Causes has made my expectations somewhat more realistic:

                                                            Thousands of years after Aristotle’s seminal work on causality, hundreds of years after Hume gave us two definitions of it, and decades after automated inference became a possibility through powerful new computers, causality is still an unsolved problem. Humans are prone to seeing causality where it does not exist and our algorithms aren’t foolproof. Even worse, once we find a cause it’s still hard to use this information to prevent or produce an outcome because of limits on what information we can collect and how we can understand it. After looking at all the cases where methods haven’t worked and researchers and policy makers have gotten causality really wrong, you might wonder why you should bother.

                                                            […]

                                                            Rather than giving up on causality, what we need to give up on is the idea of having a black box that takes some data straight from its source and emits a stream of causes with no need for interpretation or human intervention. Causal inference is necessary and possible, but it is not perfect and, most importantly, it requires domain knowledge.

                                                            Kleinberg’s book is a great general intro to causality, but it intentionally omits the mathematical details behind the various methods. I am now ready to once again go deeper into causality, perhaps starting with Kleinberg’s more technical book, Causality, Probability, and Time. Other recommendations are very welcome!

                                                            Cover image source: xkcd: Correlation

                                                            Subscribe
                                                              -
                                                              Alternatively, subscribe to RSS feed.

                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                              chewiebeans
                                                              2016-02-15 01:03:41

                                                              It seems to me that causality is another of our thought conveniences, just one more attempt at linearising our frustratingly non-linear existence, akin to teaching with Newtonian physics, segue to Einstein’s relativistic mechanics when the kids are ready (if ever). Cyclic systems can self-perpetuate in non-repeating cycles (chaos theory) but also respond with or resist change arising from external inputs. I believe when people speak of causality, what they are really thinking about (and desiring) is a conversation around stability versus volatility.

                                                              Jim Savage
                                                              2016-02-15 03:04:50

                                                              Hey Yanir - great post.

                                                              If you’ve not already, you should read Mostly Harmless Econometrics. They take quite a different approach to causality than Pearl (though there is a lot of conceptual overlap). It definitely helps build intuition for the topic. It’s also worth reading the relevant mid-70s papers from Rubin.

                                                              Yanir Seroussi
                                                              2016-02-15 08:20:06

                                                              Thanks for the pointers, Jim! I’ll check those resources out.

                                                              Will Lowe (@conjugateprior)
                                                              2016-08-02 02:32:44

                                                              It’s not a different approach. The notation is different but the two frameworks (Pearl’s and Neyman-Rubin) have been proved equivalent.

                                                              M Edward/Ed Borasky (@znmeb)
                                                              2016-02-15 10:01:19

                                                              I took a look at the Amazon sample for Causality, Probability and Time but I doubt if I’ll buy it just yet. I’ve got Judea Pearl’s Probabilistic Reasoning in Intelligent Systems already and think I want to work through that in a programming language (R is my first choice) before buying any more books. ;-)

                                                              Joanna
                                                              2016-02-15 16:01:01

                                                              I appreciate this post. I teach General Psychology, and this is a central issue that I present to my students. In the meantime, I regularly come across articles, in peer-reviewed as wells as mainstream publications, which discuss correlational data as if it were supporting a causal relationship. As I tell my students, one of the difficulties is the use of the word “factor” in both types of discussions. In correlation, factors are pieces of information which give you a more likely guess about an unknown piece of information. In causation, factors are things that contribute to something else existing. Both concepts feed the mind’s desire to find patterns in the relevant world which inform our decisions/behaviors so that we can continue living, hopefully in a pleasant state. We are often tricked by these patterns (illusions, etc.), but most of the time they pan out in a beneficial way. Making the leap from “this is how things tend to work in my immediate experience” to “this is how things work everywhere for everyone” is where theories are born, where science lives, and where we often make mistakes along the way. Proceed with caution from observation to theory, but by all means, proceed!

                                                              Kyle Gagnon
                                                              2016-02-16 01:33:53

                                                              Really great read! This is something many of my colleagues have discussed in the past. Here’s an article that might help us get closer to causality with observational data: http://goo.gl/MP7WQo, and here is a video about it: https://www.youtube.com/watch?v=uhONGgfx8Do

                                                              Boris Gorelik
                                                              2016-02-18 09:56:34

                                                              The problem with the search for causality (or, more generally, explainability) is that in many cases, it is “not interesting”. If I click on Google search results, neither me nor Google algorithm developers are truly interested how the algorithm decided to rank Page A before Page B. It is OK for me, as an end user, not to care about those details, as much as I don’t care hydraulics every time I take a shower. Is it OK for me, as a data scientist, not to care about the reasons behind my models? Honestly, I don’t yet know.

                                                              Yanir Seroussi
                                                              2016-02-18 20:42:41

                                                              I agree that in many cases the reasoning behind models isn’t interesting, as long as the models produce satisfactory results. Web search is actually a good example. Yes, many end users don’t really care how Google ranks pages, but SEO practitioners go to great lengths to understand search algorithms and get pages to rank well (see https://moz.com/search-ranking-factors for example).

                                                              As data scientists, it’s important to consider model stability in production. Sculley et al. said it well in their paper on machine learning technical debt (http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43146.pdf): “Machine learning systems often have a difficult time distinguishing the impact of correlated features. This may not seem like a major problem: if two features are always correlated, but only one is truly causal, it may still seem okay to ascribe credit to both and rely on their observed co-occurrence. However, if the world suddenly stops making these features co-occur, prediction behavior may change significantly.”

                                                              Finally, in many cases what we really care about is interventionality. I don’t think it’s a real word, but what it means is that you don’t really care whether A causes B, you want to know whether intervening to change A would change B. These inferences are critical in fields like medicine and marketing, but we can look at an example from the world of blogging, which is probably more relevant to you. Many bloggers would like to attract more readers. A possible costly intervention would be to switch platforms from WordPress to Medium. Cheaper interventions may be changing the site’s layout, writing titles that get people interested, and posting links to your content on relevant channels. Another intervention would be trying to post at different times (as implied by WordPress insights and discussed in http://yanirseroussi.com/2015/12/08/this-holiday-season-give-me-real-insights/). Obviously, one would like to apply the interventions with the highest return on investment first, and data that helps with ranking the interventions is very interesting.

                                                              Greg Gandenberger
                                                              2016-02-18 15:53:34

                                                              James Woodward’s Making Things Happen gives a fantastic, relatively non-technical analysis of causation that fits well with Pearl’s approach.

                                                              Yanir Seroussi
                                                              2016-02-18 18:56:04

                                                              Thanks! I’ll check it out.

                                                              Joel Kreager
                                                              2016-02-18 18:58:51

                                                              I’ve been thinking about this lately quite a bit. The fact that I can type this comment and send it across the internet rests on the ability to create a completely controlled causal environment. Inside the computer, all noise and randomness is kept below the threshold of the data, and every process is completely causal. Meanwhile, outside the computer, most measurements are mostly noise, and extracting any sort of causal relation is very difficult and often impossible. My mind seems to have some sort of idea of cause as something like the interaction of balls on a pool table. The que ball strikes the eight ball and knocks it into the corner pocket, etc. But when one tries to measure things, mostly one finds nothing like this. Instead, one finds that some measurements tend to be found with other measurements most of the time, but not all of the time. Cause thus seems a statistical thing, and in no way absolute. I have difficulty reconciling the two views. One thing that occured to me to investigate, was the manner in which several huge internet outages developed involving the BGP protocol. It seemed to me that every individual packet must experience a completely causal path, but the aggregate turns into the statistical causal form we most usually deal with. I haven’t followed up with this idea so far, however

                                                              Yanir Seroussi
                                                              2016-02-20 09:41:25

                                                              Interesting. I think that one of the dividing factors between traditional software engineering and data science is the attitude towards uncertainty. Whereas, as you say, coding is all about creating a controlled deterministic environment, data science and statistics thrive on uncertainty. It’s similar with computer networks as well, where there is always a non-deterministic element (e.g., packets may be lost, arrive out-of-order, or come in bursts).

                                                              Greg Gandenberger
                                                              2016-02-18 20:02:05

                                                              Causation, Prediction, and Search is also seminal (https://www.cs.cmu.edu/afs/cs.cmu.edu/project/learn-43/lib/photoz/.g/scottd/fullbook.pdf).

                                                              Disclosure: I did my PhD just down the street from the authors of Causation, Prediction, and Search, and Woodward was on my thesis committee.

                                                              Greg Gandenberger
                                                              2016-02-18 20:12:58

                                                              There is a subtle difference between Woodward’s approach and that of Pearl and of Spirtes et al., which Glymour discusses in the following places:

                                                              https://www.ncbi.nlm.nih.gov/pubmed/24887161 http://repository.cmu.edu/cgi/viewcontent.cgi?article=1280&context=philosophy

                                                              Basically, Woodward starts with the notion of an intervention on a variable and defines other concepts (e.g. direct cause) in terms of it, whereas Pearl and Spirtes et al. start with the notion of direct cause. One consequence of this difference is that properties like sex and race that cannot be intervened upon in a straightforward way cannot be causes for Woodward, strictly speaking, but can be for Pearl and Spirtes et al. This is a fine point, however, and it’s very nearly true that they simply provide alternative formulations of the same theory, with Woodward focusing on conceptual issues and the others focus on methodology.

                                                              Yanir Seroussi
                                                              2016-02-20 09:26:23

                                                              Thanks for all the pointers, Greg! I’ll definitely check them out. Personally, I have a slight bias towards Pearl, as he is my academic grandfather (he was my advisor’s advisor), but I’m keen on learning as much as possible on all the different approaches to causality. It is a fascinating area!

                                                              jasonhand24
                                                              2016-02-19 15:12:30

                                                              “Thinking, Fast & Slow” touches on some of this in later chapters. Some algebra is used to help illustrate the deception causality and efforts towards finding it can cause.

                                                              Yanir Seroussi
                                                              2016-02-19 21:44:31

                                                              Thanks! That book has been on my to-read list for a while now.

                                                              Jerome
                                                              2016-02-25 17:28:47

                                                              Great Post, Thanks Yanir. I have been afraid of being lost in Big data ie swarmed by such a vast amount of correlations. diff --git a/2016/03/20/the-rise-of-greedy-robots/index.html b/2016/03/20/the-rise-of-greedy-robots/index.html index b30d124c6..ab61bdd8e 100644 --- a/2016/03/20/the-rise-of-greedy-robots/index.html +++ b/2016/03/20/the-rise-of-greedy-robots/index.html @@ -2,7 +2,7 @@

                                                              The rise of greedy robots

                                                              March 20, 2016

                                                              Given the impressive advancement of machine intelligence in recent years, many people have been speculating on what the future holds when it comes to the power and roles of robots in our society. Some have even called for regulation of machine intelligence before it’s too late. My take on this issue is that there is no need to speculate – machine intelligence is already here, with greedy robots already dominating our lives.

                                                              Machine intelligence or artificial intelligence?

                                                              The problem with talking about artificial intelligence is that it creates an inflated expectation of machines that would be completely human-like – we won’t have true artificial intelligence until we can create machines that are indistinguishable from humans. While the goal of mimicking human intelligence is certainly interesting, it is clear that we are very far from achieving it. We currently can’t even fully simulate C. elegans, a 1mm worm with 302 neurons. However, we do have machines that can perform tasks that require intelligence, where intelligence is defined as the ability to learn or understand things or to deal with new or difficult situations. Unlike artificial intelligence, there is no doubt that machine intelligence already exists.

                                                              Airplanes provide a famous example: we don’t commonly think of them as performing artificial flight – they are machines that fly faster than any bird. Likewise, computers are super-intelligent machines. They can perform calculations that humans can’t, store and recall enormous amounts of information, translate text, play Go, drive cars, and much more – all without requiring rest or food. The robots are here, and they are becoming increasingly useful and powerful.

                                                              Who are those greedy robots?

                                                              Greed is defined as a selfish desire to have more of something (especially money). It is generally seen as a negative trait in humans. However, we have been cultivating an environment where greedy entities – for-profit organisations – thrive. The primary goal of for-profit organisations is to generate profit for their shareholders. If these organisations were human, they would be seen as the embodiment of greed, as they are focused on making money and little else. Greedy organisations “live” among us and have been enjoying a plethora of legal rights and protections for hundreds of years. These entities, which were formed and shaped by humans, now form and shape human lives.

                                                              Humans running for-profit organisations have little choice but to play by their rules. For example, many people acknowledge that corporate tax avoidance is morally wrong, as revenue from taxes supports the infrastructure and society that enable corporate profits. However, any executive of a public company who refuses to do everything they legally can to minimise their tax bill is likely to lose their job. Despite being separate from the greedy organisations we run, humans have to act greedily to effectively serve their employers.

                                                              The relationship between greedy organisations and greedy robots is clear. Much of the funding that goes into machine intelligence research comes from for-profit organisations, with the end goal of producing profit for these entities. In the words of Jeffrey Hammerbacher: The best minds of my generation are thinking about how to make people click ads. Hammerbacher, an early Facebook employee, was referring to Facebook’s business model, where considerable resources are dedicated to getting people to engage with advertising – the main driver of Facebook’s revenue. Indeed, Facebook has hired Yann LeCun (a prominent machine intelligence researcher) to head its artificial intelligence research efforts. While LeCun’s appointment will undoubtedly result in general research advancements, Facebook’s motivation is clear – they see machine intelligence as a key driver of future profits. They, and other companies, use machine intelligence to build greedy robots, whose sole goal is to increase profits.

                                                              Greedy robots are all around us. Advertising-driven companies like Facebook and Google use sophisticated algorithms to get people to click on ads. Retail companies like Amazon use machine intelligence to mine through people’s shopping history and generate product recommendations. Banks and mutual funds utilise algorithmic trading to drive their investments. None of this is science fiction, and it doesn’t take much of a leap to imagine a world where greedy robots are even more dominant. Just like we have allowed greedy legal entities to dominate our world and shape our lives, we are allowing greedy robots to do the same, just more efficiently and pervasively.

                                                              Will robots take your job?

                                                              The growing range of machine intelligence capabilities gives rise to the question of whether robots are going to take over human jobs. One salient example is that of self-driving cars, that are projected to render millions of professional drivers obsolete in the next few decades. The potential impact of machine intelligence on jobs was summarised very well by CGP Grey in his video Humans Need Not Apply. The main message of the video is that machines will soon be able to perform any job better or more cost-effectively than any human, thereby making humans unemployable for economic reasons. The video ends with a call to society to consider how to deal with a future where there are simply no jobs for a large part of the population.

                                                              Despite all the technological advancements since the start of the industrial revolution, the prevailing mode of wealth distribution remains paid labour, i.e., jobs. The implication of this is that much of the work we do is unnecessary or harmful – people work because they have no other option, but their work doesn’t necessarily benefit society. This isn’t a new insight, as the following quotes demonstrate:

                                                              • “Most men appear never to have considered what a house is, and are actually though needlessly poor all their lives because they think that they must have such a one as their neighbors have. […] For more than five years I maintained myself thus solely by the labor of my hands, and I found that, by working about six weeks in a year, I could meet all the expenses of living.” – Henry David Thoreau, Walden (1854)
                                                              • “I think that there is far too much work done in the world, that immense harm is caused by the belief that work is virtuous, and that what needs to be preached in modern industrial countries is quite different from what always has been preached. […] Modern technique has made it possible to diminish enormously the amount of labor required to secure the necessaries of life for everyone. […] If, at the end of the war, the scientific organization, which had been created in order to liberate men for fighting and munition work, had been preserved, and the hours of the week had been cut down to four, all would have been well. Instead of that the old chaos was restored, those whose work was demanded were made to work long hours, and the rest were left to starve as unemployed.” – Bertrand Russell, In Praise of Idleness (1932)
                                                              • “In the year 1930, John Maynard Keynes predicted that technology would have advanced sufficiently by century’s end that countries like Great Britain or the United States would achieve a 15-hour work week. There’s every reason to believe he was right. In technological terms, we are quite capable of this. And yet it didn’t happen. Instead, technology has been marshaled, if anything, to figure out ways to make us all work more. In order to achieve this, jobs have had to be created that are, effectively, pointless. Huge swathes of people, in Europe and North America in particular, spend their entire working lives performing tasks they secretly believe do not really need to be performed. The moral and spiritual damage that comes from this situation is profound. It is a scar across our collective soul. Yet virtually no one talks about it.” – David Graeber, On the Phenomenon of Bullshit Jobs (2013)

                                                              This leads to the conclusion that we are unlikely to experience the utopian future in which intelligent machines do all our work, leaving us ample time for leisure. Yes, people will lose their jobs. But it is not unlikely that new unnecessary jobs will be invented to keep people busy, or worse, many people will simply be unemployed and will not get to enjoy the wealth provided by technology. Stephen Hawking summarised it well recently:

                                                              If machines produce everything we need, the outcome will depend on how things are distributed. Everyone can enjoy a life of luxurious leisure if the machine-produced wealth is shared, or most people can end up miserably poor if the machine-owners successfully lobby against wealth redistribution. So far, the trend seems to be toward the second option, with technology driving ever-increasing inequality.

                                                              Where to from here?

                                                              Many people believe that the existence of powerful greedy entities is good for society. Indeed, there is no doubt that we owe many beneficial technological breakthroughs to competition between for-profit companies. However, a single-minded focus on profit means that in many cases companies do what they can to reduce their responsibility for harmful side-effects of their activities. Examples include environmental pollution, multinational tax evasion, and health effects of products like tobacco and junk food. As history shows us, in truly unregulated markets, companies would happily utilise slavery and child labour to reduce their costs. Clearly, some regulation of greedy entities is required to obtain the best results for society.

                                                              With machine intelligence becoming increasingly powerful every day, some people think that to produce the best outcomes, we just need to wait for robots to be intelligent enough to completely run our lives. However, as anyone who has actually built intelligent systems knows, the outputs of such systems are strongly dependent on the inputs and goals set by system designers. Machine intelligence is just a tool – a very powerful tool. Like nuclear energy, we can use it to improve our lives, or we can use it to obliterate everything around us. The collective choice is ours to make, but is far from simple.

                                                              Subscribe
                                                                -
                                                                Alternatively, subscribe to RSS feed.

                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                SoftwareAsLife (@SoftDevLife)
                                                                2016-08-17 11:02:44

                                                                Yes, the world has always been greedy. This reminds me of Dijkstra greedy algorithm which is used to find the shortest route. There is a lot of “steps” for an organization to become profitable. Greediness tries to find the most cost-efficient way to achieve the goal of being profitable. Let us assume that each road is a railway and trains traverse to their destinations. Each decision path will sacrifice other trains waiting to cross to their destination. If human stupidity does not overrule again, our scarce resource ultimately will be constrained by economics to one element only: time. Where do we want humans to allocate spending their time on?

                                                                Greediness will always thrive in the sense it is seen as a trait of growth by society. War, which in today’s society we ultimately condemn, was viewed in the past as one way for one nation to gain growth. Growth was limited to the domain of a specific country and the rest treated as an enemy. The end of the war was enforced by the right of not interfering other one’s own property. People had to find other means to gain growth. Thus, the concept of greediness was enforced. Greediness is using emotional appeal to manipulate other’s people habits to a specific domain.

                                                                The problem with greediness is whether people evaluate the emotional appeal matching to something positive or not positive to self and society. The maximum capacity of getting that right is:

                                                                1. The effort of people on having a multi-disciplinary knowledge of multiple domains
                                                                2. The effort of people of using their knowledge to their daily decisions.

                                                                Most consumers are passive on the above two points due to society constraints. More specifically, if people focus learning from other domains, they have the risk of underperforming on their main domain having a competitive disadvantage on their prospect of their career. This limited domain in bayesian terms makes people have low confidence for many topics leaving others to influence our decision making. I think that low confidence is the main causation we see a high rise trend where people’s decisions rely more upon push messages instead of pull messages. I haven’t seen to this day sophisticated push messages where the user has a choice of options what to see except in the on boarding phase of a product. In addition, the on boarding phase are only reasons why you should use me. There will never be a phase on reasons when not to use me. If a specific domain of a product could say to the user based on the personalized information it gathered: “Hey, you shouldn’t be using me in this situation. Use Bob instead, it will make your life easier” (This will be possible with the evolution of data). The problem is a specific domain will never explain alternative domains that can solve a user’s individual problem better because there is not a commission fee of recommending one user to another domain with qualitative information. This causes a domain which consists a set of employees to not have an interest in researching alternative domains that can solve a specific problem better because the current system has not placed a platform to reward it with a commission fee. Instead, the only way for a specific domain to thrive is by copying others ideas or owning them through acquisitions. This demotivates innovation in great sum. So far, it is only people with consciousness, with value or no value, such as start-up entrepreneurs that leave old positions and people who contribute in open source correspondingly, that go the extra mile to innovate. My whole hypothesis is that our natural instincts are a machine learner, and our only task is to do progress on everything, even our own personal life.

                                                                If those two points happen, the rule of greediness will be overruled. People will consciously evaluate whether that emotional appeal makes sense in the big picture because their jobs will force them to associate their domain with alternatives to gain a commission fee. That will gain them a more robust interdisciplinary domain knowledge causing them to have more confidence on pulling than pushing information of other domains they start to know about. Passive consumers will be less passive. The value was based before by war, now greediness, later it will be all about evaluation.

                                                                Your point of people doing less work emphasizes a more passive society than it already is. I do not propose that as that will make our situation worse. The problem is the type of tasks people do, not the task itself. People need to do tasks that progress our society instead of being passive like the game of civilization. It is the only way that makes us happy and has a purpose. Like we humans create machine learning instances have an end goal purpose, so we as humans are machine learners for a purpose where we can handle any situation that becomes a problem. Our starter pack was human suffering, hunger, and death to solve problems. Now it becomes less so and we have to be motivated by it beyond extrinsic rewards.

                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/index.html b/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/index.html index 84808a565..2e509ad6c 100644 --- a/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/index.html +++ b/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/index.html @@ -13,7 +13,7 @@ https://yanirseroussi.com/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/austin-bradford-hill.jpg 1131w," src=https://yanirseroussi.com/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/austin-bradford-hill_huf628c6f0f9f49f20b40d6213fb032a87_561648_800x0_resize_q75_box.jpg alt="Austin Bradford Hill" width=150 loading=lazy>

                                                                Austin Bradford Hill

                                                                Hill: Testing untested assumptions

                                                                To the best of my knowledge, all causal inference methods rely on untested assumptions. Specifically, we can never include all the variables in the universe in our models. Therefore, any conclusions drawn are reliant on deciding what, when, and how to measure potential causes and effects. Another issue is that no matter how good and believable our modelling is, we cannot use causal inference to convince unreasonable people. For example, some people may cite divine intervention as an unmeasurable cause of anything and everything. In addition, people with certain commercial interests often try to raise doubt about well-established causal mechanisms by making unreasonable claims for evidence of various hidden factors. For example, tobacco companies used to claim that both smoking and lung cancer were caused by a common hidden factor, making the link between smoking and lung cancer a mere association.

                                                                Assuming that we are dealing with reasonable people, there’s still the question of where we should get our untested assumptions from. This question is fairly old, and has been partly answered in 1965 by Austin Bradford Hill, with nine criteria that he recommended should be considered before calling an association causal:

                                                                1. Strength: How strong is the association? For example, lung cancer deaths of heavy smokers are 20-30 times greater than those of non-smokers.
                                                                2. Consistency: Has the association been repeatedly observed in various circumstances? For example, many different populations have exhibited an association between smoking rates and cancer.
                                                                3. Specificity: Can we pin down specific instances of the effect to specific instances of the cause? Hill sees this as a nice-to-have condition rather than a must-have – cases with multiple possible causes may not fulfil the specificity requirement.
                                                                4. Temporality: Do we know that c leads to e or are we observing them together? This is a condition that isn’t always easy to fulfil, especially when dealing with feedback loops and slow processes.
                                                                5. Biological gradient: Hill’s focus was on medicine, and this condition refers to the association exhibiting some dose-response curve. This can be generalised to other fields, as we can expect some regularity in the effect if it is a function of the cause (though it doesn’t have to be a linear function).
                                                                6. Plausibility: Do we know of a mechanism that can explain how the cause brings about the effect?
                                                                7. Coherence: Does the association conflict with our current knowledge? Even if it does, it isn’t enough to rule out causality, as our current knowledge may be incomplete or wrong.
                                                                8. Experiment: If possible, running controlled experiments may yield very powerful evidence in favour of causation.
                                                                9. Analogy: Do we know of any similar cause-and-effect relationships?

                                                                Hill summarises the list of criteria (or viewpoints) with the following statements.

                                                                Here then are nine different viewpoints from all of which we should study association before we cry causation. What I do not believe – and this has been suggested – is that we can usefully lay down some hard-and-fast rules of evidence that must be obeyed before we accept cause and effect. None of my nine viewpoints can bring indisputable evidence for or against the cause-and-effect hypothesis and none can be required as a sine qua non. What they can do, with greater or less strength, is to help us to make up our minds on the fundamental question – is there any other way of explaining the set of facts before us, is there any other answer equally, or more, likely than cause and effect?

                                                                No formal tests of significance can answer those questions. Such tests can, and should, remind us of the effects that the play of chance can create, and they will instruct us in the likely magnitude of those effects. Beyond that they contribute nothing to the ‘proof’ of our hypothesis.

                                                                Hill then goes on to criticise the increased focus on statistical significance as a condition for accepting scientific papers for publication. Remembering that this was over 50 years ago, it is a bit worrying that it has taken so long for the statistical community to formally acknowledge the fact that statistical significance does not imply scientific importance, or constitutes enough evidence to support a causal hypothesis.

                                                                Closing thoughts

                                                                This post has only scratched the surface of the vast field of study of causality. At this point, I feel like I’ve read quite a bit, and it is time to apply what I learned to real problems. I encounter questions of causality in my everyday work, but haven’t fully applied formal causal inference to any problem yet. My view is that everyone needs to at least be aware of the need to consider causality, and of what it’d take to truly prove causal impact. A large proportion of what many people need in practice may be addressed by Hill’s criteria, rather than by formal methods for causal analysis. Nonetheless, I will report back when I get a chance to apply formal causal inference to real datasets. Stay tuned!

                                                                Subscribe
                                                                  -
                                                                  Alternatively, subscribe to RSS feed.

                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                  James Savage
                                                                  2016-05-15 21:49:50

                                                                  Interesting point on the causal significance. How does this work when you have confounders in x? I’d have thought that x must contain the set of prima facie causes for which we have true exogenous variation.

                                                                  Also, how does it work when you have bad controls in x (where x includes post-treatment causes that are plausibly varied by c)?

                                                                  Yanir Seroussi
                                                                  2016-05-16 21:29:51

                                                                  Good questions :)

                                                                  To be honest, I’m not completely sure it works in all these cases, as there is always a need for interpretation to decide whether the identified causes are genuine. I tried playing a bit with the toy data from Pearl’s report on Simpson’s Paradox, but the results are not entirely convincing. However, I’m also not fully convinced that Pearl’s solution fully resolves Simpson’s Paradox, and Kleinberg does go through a few scenarios where her approach doesn’t work in her book, so I’d say that there are still quite a few open problems in the area.

                                                                  Post-treatment causes are partly addressed by the definition in Huang and Kleinberg (2015), where significance is weighted by the number of timepoints where e follows c. Again, that definition doesn’t handle all cases, but I think it’s an interesting line of research. I would definitely like to see their results reproduced by other researchers and expanded to other datasets, though.

                                                                  Jose Magana
                                                                  2016-05-17 01:19:40

                                                                  Excellent article! It has been very useful to understand what the topic of causality is about and triggered my interest to continue learning more!

                                                                  Rachel Lynne Wilkerson
                                                                  2016-12-08 11:26:40

                                                                  Thanks for this post! I share your troubles over Pearl/time/feedback loops!

                                                                  rs
                                                                  2017-03-23 06:57:16

                                                                  Nice post. have you had any chance to apply them on real datasets. Please share those results

                                                                  ingorohlfing
                                                                  2018-12-30 19:33:12

                                                                  Great post. I did not know about Kleinberg and Hill’s work. I knew a similar list of criteria from this article, which is much younger https://doi.org/10.1177%2F0951629805050859 Regarding Kleinberg: Adding time certainly is valuable, but doesn’t the smoking example change the research question from whether smoking causes lung cancer to when it causes lung cancer? The latter question is more informative and implies the former, but I’d say it is fine to ask the first question when one is not interested in the time of occurrence of cancer.

                                                                  Yanir Seroussi
                                                                  2018-12-31 01:25:05

                                                                  Thank you! I agree that the latter question is more informative, but I now think that saying that “smoking causes cancer” isn’t particularly meaningful, as it ignores both timing and dosage. A good summary of the case for well-defined interventions was provided by Miguel Hernán in this paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5207342/

                                                                  locksley1
                                                                  2021-01-02 11:44:23

                                                                  The limits of Pearl’s theory on feedback loops bothers me too. However, have you studied much Control Theory? Or dynamical systems in general? It explicitly deals with feedback loops. I’d be keen to get your thoughts on the comparison of Control Theory vs Pearl’s Causal Inference.

                                                                  Yanir Seroussi
                                                                  2021-01-03 01:07:36

                                                                  Thanks for the comment! No, I haven’t studied Control Theory. Maybe I’ll look into it one day. :)
                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2016/06/19/making-bayesian-ab-testing-more-accessible/index.html b/2016/06/19/making-bayesian-ab-testing-more-accessible/index.html index d842d0b92..45ba5e427 100644 --- a/2016/06/19/making-bayesian-ab-testing-more-accessible/index.html +++ b/2016/06/19/making-bayesian-ab-testing-more-accessible/index.html @@ -14,7 +14,7 @@ https://yanirseroussi.com/2016/06/19/making-bayesian-ab-testing-more-accessible/calcualtor-recommendation-example.png 464w," src=https://yanirseroussi.com/2016/06/19/making-bayesian-ab-testing-more-accessible/calcualtor-recommendation-example.png alt="Calculator recommendation example" loading=lazy>

                                                                  The full results also include plots of the distributions and their high density intervals. I’m pretty happy with the richer information provided by the calculator, though it still has some limitations and areas that can be improved.

                                                                  Limitations and potential improvements

                                                                  As mentioned above, I’d love to reduce the wordiness of the calculator while keeping it self-contained, but I need some feedback to understand if any explanations are redundant. It’d also be great to reduce the reliance on magic numbers, such as the 95% HDI and 0.8 precision used for generating a recommendation. However, making these settable by users would increase the complexity of using the calculator, which is already harder to use than the frequentist alternative. Nonetheless, it’s important to remember that oversimplification is the reason why it’s easier to make the wrong decision when following the classical approach.

                                                                  Other potential changes include switching to a closed-form formula rather than draws from a distribution, comparing more than two variants, and improving Kruschke’s stopping rules by simulating more scenarios than those considered in his post. In addition, I’d like to go beyond binary responses (success/failure) to support continuous rewards (e.g., revenue), and allow users to specify different costs for the variants (e.g., implementing B may cost more than sticking with A).

                                                                  Finally, it is important to keep in mind that significance testing can’t tell you whether your sample is representative of the population. For example, if you run an experiment on a very popular website, you can get a sample of thousands of people within a few minutes. Concluding an experiment based on such a sample is probably a bad idea, as it is plausible that you would reach different conclusions if you kept running the experiment for a few days, to reduce the effect that the time of day has on the results. Similarly, a few days may not be enough if your user population behaves differently on weekends – you would need to run the experiment over a few weeks. This can be extended to months and years to rule out seasonal effects, but it is up to the experimenter to weigh the practicality of considering such factors versus the need to make decisions (see articles by Peep Laja, Martin Goodson, Sam Ju, and Kohavi et al. for more details). The main thing to remember is that you just cannot completely eliminate uncertainty and the need to consider background knowledge, which is why I believe that helping more people follow the Bayesian approach is a step in the right direction.

                                                                  Subscribe
                                                                    -
                                                                    Alternatively, subscribe to RSS feed.

                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                    John Chew
                                                                    2016-06-20 00:23:50

                                                                    My hunch is that the % of HDI chosen is of less interest to a user than seeing how each test iteration alters the HDI and shifts the level of overlap between HDI and ROPE toward one of either outcome. In the example given above, would a fair interpretation be that the differences appear weighted more toward the negative than the positive? With precision, shouldn’t it be made a function of the minimum effect requested? Larger ROPE require less precision and vice versa?

                                                                    Yanir Seroussi
                                                                    2016-06-20 12:37:25

                                                                    Thanks for your comment, John! I think that it appears weighted more towards the negative because the beta distribution is symmetric when the mean is 0.5 (alpha = beta), and asymmetric in other cases, making it less pointy. According to Kruschke’s simulations, using the precision stopping rule makes the success rate estimate closer to the true mean of the underlying distribution than with other stopping rules, which tend to overestimate the success rate. I’m not sure we’d get the same results if precision were a function of the minimum effect, but I’d like to run more simulations to get a better feeling for how it works.

                                                                    Сергей Филиппов
                                                                    2017-03-23 18:11:43

                                                                    Could you tell me please how do you calculate HDI and ROPE? I am trying to replicate this calculator in R. Thanks!

                                                                    Yanir Seroussi
                                                                    2017-03-31 08:00:23

                                                                    The source code for the calculation is here: https://github.com/yanirs/yanirs.github.io/blob/master/tools/split-test-calculator/src/bayes.coffee#L139 – it shouldn’t be too hard to translate to R.

                                                                    Sam Gil
                                                                    2017-03-24 17:23:47

                                                                    Thanks for the post!

                                                                    I’m more of a business stakeholder simply trying to improve our testing practices, rather than a data scientist who understands the theories at a detailed level.

                                                                    I’m a bit confused why, if I enter the default example in your calculator (5000 trials each, 100 successes vs 130), the recommendation is to implement EITHER variant.

                                                                    Whereas, using a tool such as the following suggests a 97.8% chance the variant with 130 successes will outperform the control: https://abtestguide.com/bayesian/

                                                                    This calculator also seems to suggest the 130 successes variant should be chosen, not EITHER, as there is 95% confidence the result is not due to chance : https://abtestguide.com/calc/

                                                                    A secondary question is, if there is no predetermined sample size with the Bayesian approach, how do you plan how long to run the test for? Mainly to deal with stakeholder communication & project planning, but also to avoid peaking.

                                                                    Many thanks, diff --git a/2016/08/04/is-data-scientist-a-useless-job-title/index.html b/2016/08/04/is-data-scientist-a-useless-job-title/index.html index 6925655c5..18a0a715b 100644 --- a/2016/08/04/is-data-scientist-a-useless-job-title/index.html +++ b/2016/08/04/is-data-scientist-a-useless-job-title/index.html @@ -8,7 +8,7 @@ https://yanirseroussi.com/2016/08/04/is-data-scientist-a-useless-job-title/science-versus-engineering.png 581w," src=https://yanirseroussi.com/2016/08/04/is-data-scientist-a-useless-job-title/science-versus-engineering.png alt="Information flow in science and engineering" loading=lazy>

                                                                    Why Data Scientist is a useless job title

                                                                    Given that a data scientist is someone who does data analysis, and/or a scientist, and/or an engineer, what does it mean for a person to hold a Data Scientist position? It can mean anything, as it depends on the company and industry. A job title like Data Scientist at Company is about as meaningful as Engineer at Organisation, Scientist at Institution, or Doctor at Hospital. It gives you a general idea what the person’s background is, but provides little clue as to what the person actually does on a day-to-day basis.

                                                                    Don’t believe me? Let’s look at a few examples. Noah Lorang (Basecamp) is OK with mostly doing arithmetic. David Robinson (Stack Overflow) builds machine learning features and internal R packages, and visualises data. Robert Chang (Twitter) helps surface product insights, create data pipelines, run A/B tests, and build predictive models. Rob Hyndman (Monash University) and Jake VanderPlas (University of Washington) are academic data scientists who contribute to major R and Python open-source libraries, respectively. From personal knowledge, data scientists in many Australian enterprises focus on generating reports and building dashboards. And in my current role at Car Next Door I do a little bit of everything, e.g., implement new features, fix bugs, set up data pipelines and dashboards, run experiments, build predictive models, and analyse data.

                                                                    To be clear, the work done by many data scientists is very useful. The number of decisions made based on arbitrary thresholds and some means multiplied together on a spreadsheet can be horrifying to those of us with minimal knowledge of basic statistics. Having a good data scientist on board can have a transformative effect on a business. But it’s also very easy to end up with ineffective hires working on low-impact tasks if the business has no idea what their data scientists should be doing. This situation isn’t uncommon, given the wide range of activities that may be performed by data scientists, the lack of consensus on the definition of the field, and a general disagreement over who deserves to be called a real data scientist. We need to move beyond the hype towards clearer definitions that would help align the expectations of data scientists with those of their current and future employers.

                                                                    It’s time to specialise

                                                                    Four years ago, I changed my LinkedIn title from software engineer with a research background to data scientist. Various offers started coming my way, and they haven’t stopped since. Many people have done the same. To be a data scientist, you just need to call yourself a data scientist. The dilution of the term means that as a job title, it is useless. Useless terms are unlikely to last, so if you’re seriously thinking of becoming a data scientist, you should also consider specialising. I believe we’ll see the emergence of new specific titles, such as Machine Learning Engineer. In addition, less “sexy” titles, such as Data Analyst, may end up making a comeback. In any case, those of us who invest in building their skills, delivering value in their job, and making sure people know about it don’t have much to worry about.

                                                                    What do you think? Is specialisation inevitable or are generalist data scientists here to stay? Please let me know privately, via Twitter, or in the comments section.

                                                                    Subscribe
                                                                      -
                                                                      Alternatively, subscribe to RSS feed.

                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                      justalanm
                                                                      2016-08-05 07:45:57

                                                                      I think exactly the same, but for the moment the title is somewhat needed. I’ll make my example: I’m a data scientist because my company wants to differentiate between regular data analysts (who can’t code but are learning with me helping them) and backend software engineers who can code better than me, but lack the business knowledge and have the tendency to trow fancy algorithms at numbers without thinking about method and usefulness for the business.

                                                                      Eventually we will have new job titles, but for now we are stuck with “data scientists”. As soon as the hype will fade we’ll see people moving to new titles.

                                                                      wesleypasfield
                                                                      2016-08-09 21:33:26

                                                                      Great article - and really the ambiguity surrounding the Data Scientist title hurts everyone - Data Scientists are frustrated that they’re expected to do everything, and others are frustrated that their Data Scientists can’t do everything that they’ve heard data scientists can do. I think this will change over time as data scientists (or whatever they will be called) roles get further defined.

                                                                      Ian Miller
                                                                      2016-08-15 15:36:44

                                                                      Good article - I’ve always had a bit of problem with the term “Data Scientist” in that it infers that the person with such a title is somehow involved in research of data for scientific purposes or has a deep academic background neither of which are usually true.

                                                                      Now, what does everyone think about the term “Data Architect” - I could do a rant on that one. Suffice it to say, you are a DATABASE Architect, not a DATA architect. Data is data, it is just raw numbers. You don’t design data, you design a data model which eventually gets translated into a database. Sorry, I guess that was a bit of a rant …

                                                                      Yanir Seroussi
                                                                      2016-08-15 19:34:35

                                                                      Yeah, people like the word data more than the word database these days. There are also the various places where you can put data. You can drown it in a lake, for example…

                                                                      Jonathan Sunderland
                                                                      2016-08-31 06:49:40

                                                                      Great article, and couldn’t agree more - there’s deep irony in “Data Science” as a job title.

                                                                      I’ve started to use the term “Entrepreneurial Analyst” to be more precise about the focus on the outcome and also to allow latitude for hypotheses, exploration and discovery.

                                                                      Elena
                                                                      2016-09-08 15:13:14

                                                                      Reblogged this on codefying and commented: Especially like the antiparallel structure of scientific inquiry and engineering design.

                                                                      Bernard Clyde
                                                                      2017-02-21 21:21:37

                                                                      I find it interest that, as you said, data analysis is very useful if done by effective hires. It’s important to understand the data that you are provided with and what context it fits in. Otherwise, you could come to conclusions that miss the mark. It is important to have those who are properly qualified analyze data accurately.

                                                                      aileenscott604
                                                                      2017-07-17 11:47:15

                                                                      Great article! I also feel that anybody with an experience of 5 years and more in Data Science, can be considered for a Data Scientist role. Professionals with lesser experience can always have their roles as Data Analysts or Data Engineers. There are many certification programs that can provide you with relevant skill-sets, such as Hortonworks certification, Cloudera Certifications, Data Science Council of Ameria (DASCA) certifications etc.
                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/index.html b/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/index.html index 7fccfcff8..2cd65223c 100644 --- a/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/index.html +++ b/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/index.html @@ -30,7 +30,7 @@ https://yanirseroussi.com/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/sexy-hal-varian.jpg 850w," src=https://yanirseroussi.com/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/sexy-hal-varian_hu70cdaa620836d0cecb2e61d00c2468e8_62487_800x0_resize_q75_box.jpg alt="Hal Varian sexy statistician quote" loading=lazy>

                                                                      Once you’ve recognised your skill gaps, you may decide to hire a data scientist to help you get more value out of your data. However, despite the hype, data scientists are not magicians. In fact, because of the hype, the definition of data science is so diluted that some people say that the term itself has become useless. The truth is that dealing with data is hard, every organisation is somewhat different, and it takes time and commitment to get value out of data. The worst thing you can do is to hire an expensive expert to help you, and then ignore their advice when their findings are hard to digest. If you’re not ready to work with a data scientist, you might as well save yourself some money and remain in a state of blissful ignorance.

                                                                      Note: This article is not a portrayal of how things are with my current employer, Car Next Door. Views expressed are my own. In fact, if you want to work at a place where expert advice is acted on and uncertainty is seen as something to be studied rather than ignored, we’re hiring!

                                                                      Subscribe
                                                                        -
                                                                        Alternatively, subscribe to RSS feed.

                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                        Benoit Bernard
                                                                        2016-08-22 15:26:34

                                                                        Thanks Yanir for this post! Once again, you hit the nail on the head! We’re probably all guilty of doing any number of those mistakes at one point or another of our careers. And it wouldn’t surprise me that a lot of companies are doing all of those mistakes at the same time. I especially liked #6. Instead of stupidity, I would suggest that that ego is responsible for it.

                                                                        Yanir Seroussi
                                                                        2016-08-22 20:23:29

                                                                        Yeah, I think that Bertrand Russell was a bit too harsh – it’s really ignorance that often causes overconfidence rather than stupidity. And yes, I have made this mistake as well. Many things often look misleadingly simple if you don’t get into the fine details.

                                                                        Matthias Willerich
                                                                        2016-08-31 09:05:01

                                                                        Reblogged this on QA-notes and commented: All common sense, but as with many things, having it written down focusses the mind :-)
                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/index.html b/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/index.html index b0ed6699f..344f8a4eb 100644 --- a/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/index.html +++ b/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/index.html @@ -2,7 +2,7 @@

                                                                        Ask Why! Finding motives, causes, and purpose in data science

                                                                        September 19, 2016

                                                                        Some people equate predictive modelling with data science, thinking that mastering various machine learning techniques is the key that unlocks the mysteries of the field. However, there is much more to data science than the What and How of predictive modelling. I recently gave a talk where I argued the importance of asking Why, touching on three different topics: stakeholder motives, cause-and-effect relationships, and finding a sense of purpose. A video of the talk is available below. Unfortunately, the videographer mostly focused on me pacing rather than on the screen, but you can check out the slides here (note that you need to use both the left/right and up/down arrows to see all the slides).

                                                                        If you’re interested in the topics covered in the talk, here are a few posts you should read.

                                                                        Stakeholders and their motives

                                                                        Causality and experimentation

                                                                        Purpose, ethics, and my personal path

                                                                        Cover image: Why by Ksayer

                                                                        Subscribe
                                                                          -
                                                                          Alternatively, subscribe to RSS feed.

                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/index.html b/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/index.html index c8b2baa4a..d13ffdabb 100644 --- a/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/index.html +++ b/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/index.html @@ -11,7 +11,7 @@ https://yanirseroussi.com/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/test-your-models.jpg 430w," src=https://yanirseroussi.com/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/test-your-models.jpg alt="If you don't test your models, you're gonna have a bad time" loading=lazy>

                                                                          Conclusion: You’re better than that

                                                                          Accurate estimation of customer lifetime value is crucial to most businesses. It informs decisions on customer acquisition and retention, and getting it wrong can drive a business from profitability to insolvency. The rise of data science increases the availability of statistical and scientific tools to small and large businesses. Hence, there are few reasons why a revenue-generating business should rely on untested customer value formulas rather than on more realistic models. This extends beyond customer value to nearly every business endeavour: Relying on fabrications is not a sustainable growth strategy, there is no way around learning how to be intelligently driven by data, and no amount of cheap demagoguery and misinformation can alter the objective reality of our world.

                                                                          Subscribe
                                                                            -
                                                                            Alternatively, subscribe to RSS feed.

                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                            Ralph Haygood
                                                                            2017-01-09 21:09:02

                                                                            When I started doing data science in a business setting (after years of doing quantitative genetics in academic settings), I was puzzled by the talk of “customer lifetime value”, partly due to the issues you’ve mentioned. Even with appropriate clarifications - it’s over, say, five years, not forever, etc. - it’s a peculiar quantity, in that as typically calculated, at least by the business-y people in my vicinity, it isn’t an average over a single population of customers. Instead, it’s average, say, first-month net present value over customers who’ve been around for at least one month (or maybe over all such customers who’ve also been around for at most one year, to reduce the influence of customer behavior farther in the past, when the product catalog, marketing strategies, etc. were different), plus average second-month net present value over customers who’ve been around for at least two months, etc., that is, it’s a sum of averages over a sequence of populations of customers (which may not even be nested). And there can be further subtleties. For example, in the context of a “freemium” service such as the one that is my primary client at present, sometimes people want to measure time from when a customer signs up for an account, whereas other times people want to measure time from when a customer first buys something, which may be much later. Altogether, I’ve found that “customer lifetime value” generally requires a good deal of explanation.

                                                                            “no amount of cheap demagoguery and misinformation can alter the objective reality of our world.”: Alas, that isn’t quite true. Next week, the objective reality of how the USA is governed will be altered substantially, partly due to blatant demagoguery and misinformation.

                                                                            Jay
                                                                            2017-01-10 20:00:53

                                                                            Great analysis Yanir!

                                                                            Yanir Seroussi
                                                                            2017-01-10 20:24:33

                                                                            Thanks Ralph! I meant the last sentence in the sense of “and yet it moves”. People’s actions and choices are definitely affected by demagoguery and misinformation, but the spread of misinformation doesn’t change reality by itself. For example, Trump et al.’s climate science denialism isn’t going to alter the reality of anthropogenic climate change, though their actions are probably going to accelerate it.

                                                                            Çağrı Sarıgöz
                                                                            2017-01-12 16:14:34

                                                                            This is why Investment Banking and Venture Capital firms should hire Data Scientists.

                                                                            I think your post and the links you share can have a part on the Google search results as well in near future :)

                                                                            Ben
                                                                            2017-04-24 01:53:40

                                                                            Great post.

                                                                            There’s also the BTYD package in R that I’ve seen be used for CLV calculations although I don’t know if it could be used for anything industrial. All credit for this knowledge goes to Dan McCarthy, who just put out some great research on using CLV in non-contractual settings.

                                                                            Eleni M
                                                                            2018-02-01 13:58:15

                                                                            Hi Yanir!

                                                                            Nice post.

                                                                            How can the models you mentioned be altered in the case of a subscription based business in order to calculate the lifetime value of the customers?

                                                                            Yanir Seroussi
                                                                            2018-02-02 09:56:00

                                                                            Thanks Eleni! I think that in the case of subscription-based products, you’re better off using different models, as churn is observed and can be predicted (e.g., using a package like lifelines). Once you have an estimate of when a customer is going to churn, it’s easy to estimate their LTV (assuming constant recurring revenue). In any case, the general principle of not using closed formulas without testing their accuracy on your data still applies here.

                                                                            timothyjwhite20
                                                                            2022-07-09 16:38:07

                                                                            Thanks for the article, Yanir! I am a huge proponent of only using formulas for CLV as a starting point, used only when historical similar models aren’t available, When a good historical financial model is available, it becomes much more useful than the generic formula. I was just speaking with service vendor who was trying to convince to allow his company to perform exhaustive FMEA’s on all of our equipment when we had years of failure data to approach a maintenance strategy. Only rely on theoretical when empirical isn’t an option.
                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2017/06/03/exploring-and-visualising-reef-life-survey-data/index.html b/2017/06/03/exploring-and-visualising-reef-life-survey-data/index.html index e0a1a8bf5..0e995818f 100644 --- a/2017/06/03/exploring-and-visualising-reef-life-survey-data/index.html +++ b/2017/06/03/exploring-and-visualising-reef-life-survey-data/index.html @@ -12,7 +12,7 @@ https://yanirseroussi.com/2017/06/03/exploring-and-visualising-reef-life-survey-data/reef-life-survey-flashcards-screenshot_hu0ecf79dd903f58a2cad6274428c21648_857114_1500x0_resize_box_3.png 1500w," src=https://yanirseroussi.com/2017/06/03/exploring-and-visualising-reef-life-survey-data/reef-life-survey-flashcards-screenshot_hu0ecf79dd903f58a2cad6274428c21648_857114_800x0_resize_box_3.png alt="Reef Life Survey Flashcards screenshot" loading=lazy>

                                                                            The Flashcards tool helps users memorise the names of marine species by showing random images of species from a chosen area (RLS website | full-screen version).

                                                                            The data

                                                                            The RLS database includes data collected by volunteer scuba divers on the diversity and abundance of marine life in sites around the world. An RLS survey is performed along a 50 metre tape, which is laid at a constant depth following a reef’s contour. After laying the tape, one diver takes photos of the bottom at 2.5 metre intervals along the transect line. These photos are analysed later to classify the type of substrate or growth (e.g., hard coral or sand). Divers then complete two swims along each side of the transect. On the first swim (method 1), divers record all the fish species and large swimming animals found in a 5 metre corridor from the line. The second swim (method 2) targets invertebrates and cryptic animals, and requires keeping closer to the bottom and looking under ledges and vegetation in a 1 metre corridor from the line. The RLS manual includes all the details on how surveys are performed. The data collected in the surveys is available for download from a Data Portal hosted by the Institute for Marine and Antarctic Studies at the University of Tasmania. As of early June 2017, the downloadable dataset consists of over half a million data points from almost ten thousand surveys.

                                                                            When I first started studying marine species, I had to find a source for photos. Initially, I used Scrapy to build simple scrapers that downloaded photos from sites such as The Australian Museum, Fishbase, and Fishes of Australia. Last year, RLS made a large number of high-quality photos taken by volunteers available on their site (via the Species Search function). In addition to their high quality, an advantage of the RLS photos over images from other sources is that they were all taken in situ, i.e., in each animal’s natural habitat. On the other hand, other sites also include photos of dissections and hand-drawn illustrations, which aren’t as useful for divers who want to see marine animals as they appear in the wild. Working exclusively with the RLS image dataset has significantly improved the appearance and usefulness of the tools I built.

                                                                            The raw RLS survey data comes in the form of over 100MB of CSV files. For the purpose of building the tools, I summarised the data into two JSON files with an overall size of less than 3MB (less than 1MB when compressed). This made it possible to implement both tools as single-page apps that don’t require any requests to the server after the initial fetching of the data. The two summary JSONs are:

                                                                            • species.json – a mapping from species ID to an array of five elements: scientific name, common name, species page URL, survey method (0: method 1, 1: method 2, or 2: both), and images (array of URLs).
                                                                            • site-surveys.json – a mapping from site code to an array of seven elements: realm, ecoregion, site name, longitude, latitude, number of surveys, and species counts (mapping from each observed species ID to the number of surveys on which it was seen).

                                                                            Both files use mappings to arrays rather than nested objects to reduce the download size. I originally created the files myself by downloading the CSVs from the data portal and scraping the RLS website for images and common names. Static versions of those files from early June 2017 can be found on GitHub (species.json and site-surveys.json). As part of the integration with the RLS website, the RLS developers will implement live versions of the files, which will get updated automatically. I’ll add the links to the live versions when they become available. Please let me or the RLS team know if you find any issues with the data.

                                                                            The approach I chose to produce the species counts in site-surveys.json doesn’t take abundance into account, i.e., each species is counted once per survey regardless of the number of times it was seen on the survey. Ignoring abundance means that for sites with few surveys, the species count may not be a good indicator of future likelihood of occurrence. For example, some fish are solitary and seen rarely, while others occur in schools and are likely to be seen on every survey. However, this is less of an issue for sites with many surveys. In addition, this simple counting approach is easier to explain than some approaches that do account for abundance.

                                                                            Implementation details

                                                                            The source code for the tools can be found in my GitHub Pages repository. Each tool is a simple single-page application, consisting of three files: index.jade, main.coffee, and style.less. In addition, the root source directory contains some common code in common.less and util.coffee, as well as configuration files for npm and Grunt. Grunt is used to compile the source files from Jade/Pug, CoffeeScript, and Less to HTML, JS, and CSS respectively. These files are then served statically by GitHub Pages.

                                                                            The common CoffeeScript code loads the JSONs asynchronously, and processes them into nested mappings that are easier to work with than arrays. In addition, the common code contains a method to summarise counts from multiple sites, by aggregating them as simple sums. This means that sites that are surveyed more frequently get weighted more heavily. For example, if a certain fish X was seen once in site A, twice in site B, and never in site C, its count across A, B, and C is 1 + 2 + 0 = 3, but if A was surveyed once, B was surveyed twice, and C was surveyed seven times, X’s aggregate frequency is 3 / (1 + 2 + 7) = 30%. In the future, it may be worth normalising each site’s species counts by the number of times the site was surveyed (making X’s aggregate frequency (1 / 1 + 2 / 2 + 0 / 7) / 3 = 66.67%), but then rare species in rarely-surveyed sites may be overweighted.

                                                                            The Frequency Explorer tool uses the Google Maps API to show a map with all the past survey sites. Users can select sites by drawing an area on the map, or by searching for site names in a Select2 box. The tool fails gracefully when Google Maps isn’t available, which makes it possible to run it offline (assuming you have local copies of the species images). This was very useful on my last trip to the Coral Sea, where I was away from mobile reception for weeks. When sites are selected, the code generates a summary table of the species frequencies, which can be exported to a dynamically-generated CSV. In addition, users can choose to display images of all the species in the table. As this can trigger the download of thousands of images, I used vanilla-lazyload to only load images when they enter the viewport. Finally, Frequency Explorer can also be used as a site selector for the Flashcards tool, as it contains a link to launch Flashcards with the set of selected sites (which is passed in the Flashcards query string).

                                                                            The Flashcards tool relies on the excellent reveal.js library to dynamically generate a presentation with a random subset of images of species that were recorded at the selected sites. The presentation consists of pairs of image and name slides – each image slide is followed by a slide where the name of the previously-shown animal is revealed. As I found that trying to memorise all the species at once is too hard, I added the ability to adjust the difficulty level of the flashcards by setting a frequency threshold (e.g., show only species that were recorded on 25% of surveys), or by focusing on observations from a single survey method (e.g., method 2 surveys in the tropics tend to be much less diverse than method 1 surveys). To avoid reloading the entire page when the settings change, the slides are regenerated dynamically. Reveal isn’t really built to account for dynamic regeneration of slides, so I had to add a call to Reveal.toggleOverview(false) to get the cards to refresh correctly, but other than that it worked perfectly.

                                                                            Future work

                                                                            There are several possible extensions to the work done so far.

                                                                            First, the integration of the tools into the RLS website is incomplete. They are still served in iframes from my GitHub Pages account, and the JSON data isn’t updated automatically. Completing the integration is dependent on the RLS developers, who also have other priorities. Other RLS-dependent items include better optimisation of images (they’re currently scaled down on the client side), and general performance improvements to the site.

                                                                            Second, the tools themselves could be improved. For example, reliance on third-party libraries should be reduced (e.g., Frequency Explorer uses Bootstrap due to my limited design skills), and it’d be nice if site selections were stored and read from the URL of Frequency Explorer (this is already done for Flashcards). In addition, as the tools are used to train new RLS divers, it’d be useful to extend the Flashcards tool to run in test mode, where users would type in the names of the animals rather than just passively scroll through the presentation. This would make it possible to assess diver readiness to perform surveys based on their test scores.

                                                                            Finally, many other interesting things can be done with the RLS data (in addition to producing scientific papers and reports, which is the main focus of the researchers behind the project). Examples include using the images to automate species identification (as discussed more thoroughly in my previous post on the topic), and building models to predict survey output and detect anomalies (e.g., due to climate change or other unusual factors). If you have other ideas, or end up playing with the data and coming with interesting results, please share your findings in the comments section.

                                                                            Subscribe
                                                                              -
                                                                              Alternatively, subscribe to RSS feed.

                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/index.html b/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/index.html index 9fe7e7e5c..1df379b0f 100644 --- a/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/index.html +++ b/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/index.html @@ -2,7 +2,7 @@

                                                                              My 10-step path to becoming a remote data scientist with Automattic

                                                                              July 29, 2017

                                                                              About two years ago, I read the book The Year without Pants, which describes the author’s experience leading a team at Automattic (the company behind WordPress.com, among other products). Automattic is a fully-distributed company, which means that all of its employees work remotely (hence pants are optional). While the book discusses some of the challenges of working remotely, the author’s general experience was very positive. A few months after reading the book, I decided to look for a full-time position after a period of independent work. Ideally, I wanted a well-paid data science-y remote job with an established distributed tech company that offers a good life balance and makes products I care about. Automattic seemed to tick all my boxes, so I decided to apply for a job with them. This post describes my application steps, which ultimately led to me becoming a data scientist with Automattic.

                                                                              Before jumping in, it’s worth noting that this post describes my personal experience. If you apply for a job with Automattic, your experience is likely to be different, as the process varies across teams, and evolves over time.

                                                                              📧 Step 1: Do background research and apply

                                                                              I decided to apply for a data wrangler position with Automattic in October 2015. While data wrangler may sound less sexy than data scientist, reading the job ad led me to believe that the position may involve interesting data science work. This impression was strengthened by some LinkedIn stalking, which included finding current data wranglers and reading through their profiles and websites. I later found out that all the people on the data division start out as data wranglers, and then they may pick their own title. Some data wranglers do data science work, while others are more focused on data engineering, and there are some projects that require a broad range of skills. As the usefulness of the term data scientist is questionable, I’m not too fussed about fancy job titles. It’s more important to do interesting work in a supportive environment.

                                                                              Applying for the job was fairly straightforward. I simply followed the instructions from the ad:

                                                                              Does this sound interesting? If yes, please send a short email to jobs @ this domain telling us about yourself and attach a resumé. Let us know what you can contribute to the team. Include the title of the position you’re applying for and your name in the subject. Proofread! Make sure you spell and capitalize WordPress and Automattic correctly. We are lucky to receive hundreds of applications for every position, so try to make your application stand out. If you apply for multiple positions or send multiple emails there will be one reply.

                                                                              Having been on the receiving side of job applications, I find it surprising that many people don’t bother writing a cover letter, addressing the selection criteria in the ad, or even applying for a job they’re qualified to do. Hence, my cover letter was fairly short, comprising of several bullet points that highlight the similarities between the job requirements and my experience. It was nothing fancy, but simple cover letters have worked well for me in the past.

                                                                              ⏳ Step 2: Wait patiently

                                                                              The initial application was followed by a long wait. From my research, this is the typical scenario. This is unsurprising, as Automattic is a fairly small company with a large footprint, which is both distributed and known as a great place to work (e.g., its Glassdoor rating is 4.9). Therefore, it attracts many applicants from all over the world, which take a while to process. In addition, Matt Mullenweg (Automattic’s CEO) reviews job applications before passing them on to the team leads.

                                                                              As I didn’t know that Matt reviewed job applications, I decided to try to shorten the wait by getting introduced to someone in the data division. My first attempt was via a second-degree LinkedIn connection who works for Automattic. He responded quickly when I reached out to him, saying that his experience working with the company is in line with the Glassdoor reviews – it’s the best job he’s had in his 15-year-long career. However, he couldn’t help me with an intro, because there is no simple way around Automattic’s internal processes. Nonetheless, he reassured me that it is worth waiting patiently, as the strict process means that you end up working with great people.

                                                                              I wasn’t in a huge rush to find a job, but in December 2015 I decided to accept an offer to become the head of data science at Car Next Door. This was a good decision at the time, as I believe in the company’s original vision of reducing the number of cars on the road through car sharing, and it seemed like there would be many interesting projects for me to work on. The position wasn’t completely remote, but as the company was already spread across several cities, I was able to work from home for a day or two every week. In addition, it was a pleasant commute by bike from my Sydney home to the office, so putting the fully-remote job search on hold didn’t seem like a major sacrifice. As I haven’t heard anything from Automattic at that stage, it seemed unwise to reject a good offer, so I started working full-time with Car Next Door in January 2016.

                                                                              I successfully attracted Automattic’s attention with a post I published on the misuse of the word insights by many tech companies, which included an example from WordPress.com. Greg Ichneumon Brown, one of the data wranglers, commented on the post, and invited me to apply to join Automattic and help them address the issues I raised. This happened after I accepted the offer from Car Next Door, and hasn’t resulted in any speed up of the process, so I just gave up on Automattic and carried on with my life.

                                                                              💬 Step 3: Chat with the data lead

                                                                              I finally heard back from Automattic in February 2016 (four months after my initial application and a month into my employment with Car Next Door). Martin Remy, who leads the data division, emailed me to enquire if I’m still interested in the position. I informed him that I was no longer looking for a job, but we agreed to have an informal chat, as I’ve been waiting for such a long time.

                                                                              As is often the case with Automattic interviews, the chat with Martin was completely text-based. Working with a distributed team means that voice and video calls can be hard to schedule. Hence, Automattic relies heavily on textual channels, and text-based interviews allow the company to test the written communication skills of candidates. The chat revolved around my past work experience, and Martin also took the time to answer my questions about the company and the data division. At the conclusion of the chat, Martin suggested I contact him directly if I was ever interested in continuing the application process. While I was happy with my position at the time, the chat strengthened my positive impression of Automattic, and I decided that I would reapply if I were to look for a full-time position again.

                                                                              My next job search started earlier than I had anticipated. In October 2016, I decided to leave Car Next Door due to disagreements with the founders over the general direction of the company. In addition, I had more flexibility in choosing where to live, as my personal circumstances had changed. As I’ve always been curious about life outside the capital cities of Australia, I wanted to move away from Sydney. While I could have probably continued working remotely with Car Next Door, I felt that it would be better to find a job with a fully-distributed team. Therefore, I messaged Martin and we scheduled another chat.

                                                                              The second chat with Martin took place in early November. Similarly to the first chat, it was conducted via Skype text messages, and revolved around my work in the time that has passed since the first chat. This time, as I was keen on continuing with the process, I asked more specific questions about what kind of work I’m likely to end up doing and what the next steps would be. The answers were that I’d be joining the data science team, and that the next steps are a pre-trial test, a paid trial, and a final interview with Matt. While this sounds straightforward, it took another six months until I finally became an Automattic employee (but I wasn’t in a rush).

                                                                              ☑️ Step 4: Pass the pre-trial test

                                                                              The pre-trial test consisted of a data analysis task, where I was given a dataset and a set of questions to answer by Carly Stambaugh, the data science lead. The goal of the test is to evaluate the candidate’s approach to a problem, and assess organisational and communication skills. As such, the focus isn’t on obtaining a specific result, so candidates are given a choice of several potential avenues to explore. The open-ended nature of the task is reminiscent of many real-world data science projects, where you don’t always have a clear idea of what you’re going to discover. While some people may find this kind of uncertainty daunting, I find it interesting, as it is one of the things that makes data science a science.

                                                                              I spent a few days analysing the data and preparing a report, which was submitted as a Jupyter Notebook. After submitting my initial report, there were a few follow-up questions, which I answered by email. The report was reviewed by Carly and Martin, and as they were satisfied with my work, I was invited to proceed to the next stage: A paid trial project.

                                                                              👨‍💻 Step 5: Do the trial project

                                                                              The main part of the application process with Automattic is the paid trial project. The rationale behind doing paid trials was explained a few years ago by Matt in Hire by Auditions, Not Resumes:

                                                                              Before we hire anyone, they go through a trial process first, on contract. They can do the work at night or over the weekend, so they don’t have to leave their current job in the meantime. We pay a standard rate of $25 per hour, regardless of whether you’re applying to be an engineer or the chief financial officer.

                                                                              During the trials, we give the applicants actual work. If you’re applying to work in customer support, you’ll answer tickets. If you’re an engineer, you’ll work on engineering problems. If you’re a designer, you’ll design.

                                                                              There’s nothing like being in the trenches with someone, working with them day by day. It tells you something you can’t learn from resumes, interviews, or reference checks. At the end of the trial, everyone involved has a great sense of whether they want to work together going forward. And, yes, that means everyone — it’s a mutual tryout. Some people decide we’re not the right fit for them.

                                                                              The goal of my trial project was to improve the Elasticsearch language detection algorithm. This took about a month, and ultimately resulted in a pull request that got merged into the language detection plugin. I find this aspect of the process pretty exciting: While the plugin is used to classify millions of documents internally by Automattic, its impact extends beyond the company, as Elasticsearch is used by many other organisations and projects. This stands in contrast to many other technical job interviews, which consist of unpaid work on toy problems under stressful conditions, where the work performed is ultimately thrown away. While the monetary compensation for the trial work is lower than the market rate for data science consulting, I valued the opportunity to work on a real open source project, even if this hadn’t led to me getting hired.

                                                                              There was much more to the trial project than what’s shown in the final pull request. Most of the discussions were held on an internal project thread, primarly under the guidance of Carly (the data science lead), and Greg (the data wrangler who replied to my post a year earlier). The project was kicked off with a general problem statement: There was some evidence that the Elasticsearch language detection plugin doesn’t perform well on short texts, and my mission was to improve it. As the plugin didn’t include any tests for short texts, one of the main contributions of my work was the creation of datasets and tests to measure its accuracy on texts of different lengths. This was followed by some tweaks that improved the plugin’s performance, as summarised in the pull request. Internally, this work consisted of several iterations where I came up with ideas, asked questions, implemented the ideas, shared the results, and discussed further steps. There are still many possible improvements to the work done in the trial. However, as trials generally last around a month, we decided to end it after a few iterations.

                                                                              I enjoyed the trial process, but it is definitely not for everyone. Most notably, there is a strong emphasis on asynchronous text-based communication, which is the main mode by which projects are coordinated at Automattic. People who don’t enjoy written communication may find this aspect challenging, but I have always found that writing helps me organise my thoughts, and that I retain information better when reading than when listening to people speak. That being said, Automatticians do meet in person several times a year, and some teams have video chats for some discussions. While doing the trial, I had a video chat with Carly, which was the first (and last) time in the process that I got to see and hear a live human. However, this was not an essential part of the trial project, as our chat was mostly on the data scientist role and my job expectations.

                                                                              ⏳ Step 6: Wait patiently

                                                                              I finished working on the trial project just before Christmas. The feedback I received throughout the trial was positive, but Martin, Carly, and Greg had to go through the work and discuss it among themselves before making a final decision. This took about a month, due to the holiday period, various personal circumstances, and the data science team meetup that was scheduled for January 2017. Eventually, Martin got back to me with positive news: They were satisfied with my trial work, which meant there was only one stage left – the final interview with Matt Mullenweg, Automattic’s CEO.

                                                                              👉 Step 7: Ping Matt

                                                                              Like other parts of the process, the interview with Matt is text-based. The way it works is fairly simple: I was instructed to message Matt on Slack and wait for a response, which may take days or weeks. I sent Matt a message on January 25, and was surprised to hear back from him the following morning. However, that day was Australia Day, which is a public holiday here. Therefore, I only got back to him two hours after he messaged me that morning, and by that time he was probably already busy with other things. This was the start of a pretty long wait.

                                                                              ⏳ Step 8: Wait patiently

                                                                              I left Car Next Door at the end of January, as I figured that I would be able to line up some other work even if things didn’t work out with Automattic. My plan was to take some time off, and then move up to the Northern Rivers area of New South Wales. I had two Reef Life Survey trips planned, so I wasn’t going to start working again before mid-April. I assumed that I would hear back from Matt before then, which would have allowed me to make an informed decision whether to look for another job or not.

                                                                              After two weeks of waiting, the time for my dive trips was nearing. As I was going to be without mobile reception for a while, I thought it’d be worth letting Matt know my schedule. After discussing the matter with Martin, I messaged Matt. He responded, saying that we might as well do the interview at the beginning of April, as I won’t be starting work before that time anyway. I would have preferred to be done with the interview earlier, but was happy to have some certainty and not worry about missing more chat messages before April.

                                                                              In early April, I returned from my second dive trip (which included a close encounter with Cyclone Debbie), and was hoping to sort out my remote work situation while completing the move up north. Unfortunately, while the move was successful, I was ready to give up on Automattic because I haven’t heard back from Matt at all in April. However, Martin remained optimistic and encouraged me to wait patiently, which I did as I was pretty busy with the move and with some casual freelancing projects.

                                                                              💬 Step 9: Chat with Matt and accept the job offer

                                                                              The chat with Matt finally happened on May 2. As is often the case, it took a few hours and covered my background, the trial process, and some other general questions. I asked him about my long wait for the final chat, and he apologised for me being an outlier, as most chats happen within two weeks of a candidate being passed over to him. As the chat was about to conclude, we got to the topic of salary negotiation (which went well), and then the process was finally over! Within a few hours of the chat I was sent an offer letter and an employment contract. As Automattic has an entity in Australia (called Ausomattic), it’s a fairly standard contract. I signed the contract and started work the following week – over a year and a half after my initial application. Even before I started working, I booked tickets to meet the data division in Montréal – a fairly swift transition from the long wait for the final interview.

                                                                              🎉 Step 10: Start working and choose a job title

                                                                              As noted above, Automatticians get to choose their own job titles, so to become a data scientist with Automattic, I had to set my job title to Data Scientist. This is generally how many people become data scientists these days, even outside Automattic. However, job titles don’t matter as much as job satisfaction. And after 2.5 months with Automattic, I’m very satisfied with my decision to join the company. My first three weeks were spent doing customer support, like all new Automattic employees. Since then, I’ve been involved in projects to make engagement measurement more consistent (harder than it sounds, as counting things is hard), and to improve the data science codebase (e.g., moving away from Legacy Python). Besides that, I also went to Montréal for the data division meetup, and have started getting into chatbot work. I’m looking forward to doing more work and sharing my experience here and on data.blog.

                                                                              Subscribe
                                                                                -
                                                                                Alternatively, subscribe to RSS feed.

                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                Jonathan Wood
                                                                                2017-07-29 18:26:09

                                                                                Very enlightening post! It was very awesome to see that the insights you saw to Elasticsearch went to a PR. I bet that was worth the whole thing!

                                                                                Mostafa
                                                                                2017-07-30 09:33:44

                                                                                That’s very exciting, I wanted to ask are you a self learner or do you have a degree,can you please share your background. Thank you

                                                                                Yanir Seroussi
                                                                                2017-07-31 01:37:04

                                                                                Thanks Mostafa. Yes, I have a BSc in computer science, and a PhD in what you would now call data science. See: https://www.linkedin.com/in/yanirseroussi/

                                                                                Pravin Singh
                                                                                2017-11-23 08:51:50

                                                                                This was an amazing post, Yanir! Loved the breakdown and the patience you had for the whole process, very well played and you really deserved it! :)

                                                                                P.S: Really can connect as I’ve been working independently for a while now and would definitely be open to looking for long-term contracts or remote jobs like this.

                                                                                Baker
                                                                                2018-12-23 21:57:33

                                                                                Your post is really a therapy to most people who apply for jobs and loose hope of waiting. I believe patience is a key to everything. Thqnks
                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2017/09/02/state-of-bandcamp-recommender/index.html b/2017/09/02/state-of-bandcamp-recommender/index.html index abc59e20d..cace06669 100644 --- a/2017/09/02/state-of-bandcamp-recommender/index.html +++ b/2017/09/02/state-of-bandcamp-recommender/index.html @@ -2,7 +2,7 @@

                                                                                State of Bandcamp Recommender, Late 2017

                                                                                September 2, 2017

                                                                                November 2017: Update and goodbye

                                                                                I’ve decided to shut down Bandcamp Recommender (BCRecommender), despite hearing back from a few volunteers. The main reasons are:

                                                                                1. Bandcamp now shows album recommendations at the bottom of album pages. While this isn’t quite the same as BCRecommender, I hope that it will evolve to a more comprehensive recommender system.
                                                                                2. I tried to contact Bandcamp to get their support for the continued running of BCRecommender. I have not heard back from them. It would have been nice to receive some acknowledgement that they find BCRecommender useful.
                                                                                3. As discussed below, I don’t have much time to spend on the project, and handing it off to other maintainers would have been time-consuming. Given reasons 1 and 2, I don’t feel like it’s worth the effort. Thanks to everyone who’s contacted me – you’re awesome!

                                                                                September 2017: Original announcement

                                                                                I released the first version of Bandcamp Recommender (BCRecommender) about three years ago, with the main goal of surfacing music recommendations from Bandcamp. A secondary goal was learning more about building and marketing a standalone web app. As such, I shared a few posts about BCRecommender over the years:

                                                                                The last of the above posts was published in November 2015 – almost two years ago. Most of the work on BCRecommender was done up to that point, when my main focus was on part-time contracting while working on my own projects. However, since January 2016 I’ve mostly been working full-time, so I haven’t had the time to give enough attention to the project. Therefore, it looks like it’s time for me to say goodbye to BCRecommender.

                                                                                Despite the lack of attention, about 5,000 people still visit BCRecommender every month (down from a peak of around 9,000). I know that people find it useful, even though it hasn’t been functionally updated in a long time (though the recommendations have been refreshed a few times). In an ideal world, BCRecommender would be replaced by algorithmic recommendations from Bandcamp. But unfortunately, Bandcamp still doesn’t offer personalised recommendations. This is a shame, because such recommendations could be of great benefit to both artists and fans. Millions of tracks and albums have been published on Bandcamp, meaning that serving personalised recommendations that cover their full catalogue can only be achieved using algorithms. However, it seems like they’re not interested in building this kind of functionality.

                                                                                Rather than simply pulling the plug on BCRecommender, I thought I’d put a call out to see if anyone is interested in maintaining it. I’m happy to open source the code and hand the project over to someone else if it means it would be in good hands. With a little bit of work, BCRecommender can be turned into a full Bandcamp-based personalised radio station. If you think you’d be a good fit for maintaining the project, drop me a line and we can discuss further. If you just love BCRecommender, you can also let Bandcamp know that you want them to implement algorithmic recommendations (e.g., on Twitter or by emailing support@bandcamp.com). I’ll keep BCRecommender alive for about two more months and see if I get any responses. Either way, I’ll be saying goodbye to maintaining it before the end of the year.

                                                                                Subscribe
                                                                                  -
                                                                                  Alternatively, subscribe to RSS feed.

                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                  Shanky
                                                                                  2017-10-15 20:08:20

                                                                                  Cool…not sure why and when i subscribed to your mailing list. and now quite surprised to hear that Bandcamp Recommender was your project. i am bandcamp freak… Bandcamp has recently strarted recommendations at the bottom ;-) seems primitive though. example https://ogreyouasshole.bandcamp.com/album/crossword-lost-sigh-days-james-mcnew-remixes would love to hear about the basic logic you used behind the “recommendations” . I have no technical knowledge at all but a few years ago thought of a basic recommendation model ..but couldnt take it forward though…i thought ‘contextualizing’ artists would be a cool way to connect bands. diff --git a/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/index.html b/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/index.html index 050e2695d..6c4979ab4 100644 --- a/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/index.html +++ b/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/index.html @@ -3,7 +3,7 @@ Harrington Emerson (1911)

                                                                                  I want to become a data science freelancer. Can you provide some advice?

                                                                                  As with any freelancing job, expect to spend much of your time on sales and networking. I've only explored the freelancing path briefly, but Radim Řehůřek has published great slides on the topic. If you're thinking of freelancing as a way of gaining financial independence, also consider spending less, earning more, and investing wisely.

                                                                                  Can you recommend an academic data science degree?

                                                                                  Sorry, but I don't know much about those degrees. Boris Gorelik has some interesting thoughts on studying data science.

                                                                                  Will you be my mentor?

                                                                                  Probably not, unless you're hard-working, independent, and doing something I find interesting. Feel free to contact me if you believe we'd both find the relationship beneficial.

                                                                                  Can you help with my project?

                                                                                  Possibly. If you think I'd find your project exciting, please do contact me.

                                                                                  What about ethics?

                                                                                  What about them? There isn't a single definition of right and wrong, as morality is multi-dimensional. I believe it's important to question your own choices, and avoid applying data science blindly. For me, this means divesting from harmful industries like fossil fuels and striving to go beyond the creation of greedy robots (among other things).

                                                                                  I’m a manager. When should I hire a data scientist and start using machine learning?

                                                                                  There's a good chance you don't need a data scientist yet, but you should be aware of common pitfalls when trying to be data-driven. It's also worth reading Paras Chopra's post on what you need to know before you board the machine learning train.

                                                                                  Do you want to buy my products or services?

                                                                                  No. If I did, I'd contact you.

                                                                                  I have a question that isn’t answered here or anywhere on the internet, and I think you can help. Can I contact you?

                                                                                  Sure, use the form on this page.

                                                                                  Subscribe
                                                                                    -
                                                                                    Alternatively, subscribe to RSS feed.

                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                    Simon
                                                                                    2017-10-15 13:26:01

                                                                                    Thanks so much for sharing this Yanir!

                                                                                    Eric Colson
                                                                                    2017-10-15 16:38:31

                                                                                    Indeed, such questions seem to be very recurring. Thanks for providing answers to help guide folks. I might add a few things:

                                                                                    when ready for the job search… Advice to Data Scientists on Where to Work http://multithreaded.stitchfix.com/blog/2015/03/31/advice-for-data-scientists/

                                                                                    if you are going to get into data science, do it for the right reasons. Let your passion drive! https://www.quora.com/How-do-I-move-from-data-scientist-to-data-science-management

                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | diff --git a/2018/07/22/defining-data-science-in-2018/index.html b/2018/07/22/defining-data-science-in-2018/index.html index 6dd5d47d3..fcb637a39 100644 --- a/2018/07/22/defining-data-science-in-2018/index.html +++ b/2018/07/22/defining-data-science-in-2018/index.html @@ -2,7 +2,7 @@

                                                                                    Defining data science in 2018

                                                                                    July 22, 2018

                                                                                    I got my first data science job in 2012, the year Harvard Business Review announced data scientist to be the sexiest job of the 21st century. Two years later, I published a post on my then-favourite definition of data science, as the intersection between software engineering and statistics. Unfortunately, that definition became somewhat irrelevant as more and more people jumped on the data science bandwagon – possibly to the point of making data scientist useless as a job title. However, I still call myself a data scientist. Even better – I still get paid for being a data scientist. But what does it mean? What do I actually do here? This article is a short summary of my understanding of the definition of data science in 2018.

                                                                                    It’s not all about machine learning

                                                                                    As I was wrapping up my PhD in 2012, I started thinking about my next steps. I knew I wanted to get back to working in the tech industry, ideally with a small startup. But it wasn’t clear to me how to market myself – my LinkedIn title at the time was “software engineer with a research background”, which is a bit of a mouthful. Around that time I heard about Kaggle and decided to try competing. This went pretty well, and exposed me to the data science community globally and in Melbourne, where I was living at the time. That’s how I first met Adam Neumann, the founder of Giveable, a startup that aimed to recommend gifts based on social networking data. Upon graduating, I joined Giveable as a data scientist. Changing my LinkedIn title quickly led to many other offers, but I was happy to be working on Giveable – I felt fortunate to have found a startup job that was related to my PhD research on recommender systems.

                                                                                    My understanding of data science at the time was heavily influenced by Kaggle and the tech industry. Kaggle was only about predictive modelling competitions back then, and so I believed that data science is about using machine learning to build models and deploy them as part of various applications. I was very comfortable with that definition, having spent my PhD years on several predictive modelling tasks, and having worked as a software engineer prior to that.

                                                                                    Things have changed considerably since 2012. It is now much easier to deploy machine learning models, even without a deep understanding of how they work. Many more people call themselves data scientists, including some who are more focused on data analysis than on building data products. Even Kaggle – which is now owned by Google – has broadened its scope beyond modelling competitions to support other types of analysis. Numerous articles have been published on the meaning of data science in the past six years. We seem to be going towards a broad definition of the field, which includes any type of general data analysis. This trend of broadening the definition may make data scientist somewhat useless as a job title. However, I believe that data science tasks remain useful, as shown by the following definitions.

                                                                                    Recent definitions by Hernán, Hawkins, and Dubossarsky

                                                                                    In a recent article, Hernán et al. classify data science tasks into three types: description, prediction, and causal inference. Like other authors, they argue that causal inference has been neglected by traditional statistics and some scientific disciplines. They claim that the emergence of data science is an opportunity to get causal inference “right”. Further, they emphasise the importance of domain expert knowledge, which is essential in causal inference. Defining data science in this broad manner seems to capture the essence of what the field is about these days. However, purely descriptive tasks are still often performed by data analysts rather than scientists. And the distinction between prediction and causal inference can be a bit fuzzy, especially as the tools for the latter are at a lower level of maturity. In addition, while I agree with Hernán et al. that domain expertise is important, it seems unlikely that this will forever be the case. No one is born an expert – expertise is gained by learning from and interacting with the world. Therefore, it’s plausible that gaining expertise can and will be automated. Further, there are numerous cases where experts were proven to be wrong. For example, it wasn’t so long ago that doctors recommended smoking.

                                                                                    Despite the importance of domain knowledge, one can argue that scientists that specialise in a single domain are not data scientists. In fact, the ability to go beyond one domain and think of data in a more abstract manner is what makes a data scientist. Applying this abstract knowledge often requires some domain expertise or input from domain experts, but most data science techniques are not domain-specific – they can be applied to many different problems. John Hawkins explains this point well in an article titled why all scientists are not data scientists:

                                                                                    Those scientists and statisticians who have focused themselves on understanding the limitations and possibilities of making inferences from experimental data are the ones who are the forerunners to data scientists. They have a skill which transcends the particulars of what it takes to do lab work on cell cultures, or field studies for ecology etc. Their core skill involves thinking about the data involved at an abstracted level. To ask the question “given data with these properties, what conclusions can we draw?”

                                                                                    Finally, according to Eugene Dubossarsky, “there’s only one purpose to data science, and that is to support decisions. And more specifically, to make better decisions. That should be something no one can argue with.” This goal-focused definition is unsurprising, given the fact that Eugene runs a training and consulting business and has been working in the field for over 20 years. I’m not going to argue with him, but to put it all together, we can define data science as a field that deals with description, prediction, and causal inference from data in a manner that is both domain-independent and domain-aware, with the ultimate goal of supporting decisions.

                                                                                    What about AI?

                                                                                    Everyone loves a good buzzword, and these days AI (Artificial Intelligence) is one of the hottest buzzwords. However, despite what some people may try to tell you, AI is unlikely to make data science obsolete any time soon. Following the above definition, as long as there is a need to make decisions based on data, there will be a need for data scientists. This includes decisions that aren’t made by humans, as data scientists are involved in building systems that make decisions autonomously.

                                                                                    The resurgence of AI feels somewhat amusing given my personal experience. One of the reasons I decided to pursue a PhD in natural language processing and personalisation was my interest in what I considered to be AI back in 2008. My initial introduction to the field was through an AI course and a project I did as part of my bachelor’s degree in computer science. However, by the time I graduated from my PhD, saying that I’m an AI expert seemed less useful than calling myself a data scientist. It may be that the field is about to shift again, and that rebranding as an AI expert would be more beneficial (though I’d be doing exactly the same work). Titles are somewhat silly – I’m going to continue working with data to support decisions for as long as there is demand for this kind of work and I continue enjoying it. There is plenty to learn and develop in this area, regardless of buzzwords and sexy titles.

                                                                                    Subscribe
                                                                                      -
                                                                                      Alternatively, subscribe to RSS feed.

                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                      Pravin
                                                                                      2018-07-24 06:29:02

                                                                                      Great set of definitions and path of evolutions here!

                                                                                      There has to be chaos and confusion as it evolves surely, but the consensus as you very well mentioned is decisions. Anything done in the data world, if not leading to decisions is not quite viable in long term.

                                                                                      Thanks for sharing your thoughts, love reading your blog.

                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2018/11/03/reflections-on-remote-data-science-work/index.html b/2018/11/03/reflections-on-remote-data-science-work/index.html index 6b5ddbe59..4c8b8b8eb 100644 --- a/2018/11/03/reflections-on-remote-data-science-work/index.html +++ b/2018/11/03/reflections-on-remote-data-science-work/index.html @@ -2,7 +2,7 @@

                                                                                      Reflections on remote data science work

                                                                                      November 3, 2018

                                                                                      It’s been about a year and a half since I joined Automattic as a remote data scientist. This is the longest I’ve been in one position since finishing my PhD in 2012. This is also the first time I’ve worked full-time with a fully-distributed team. In this post, I briefly discuss some of the top pluses and minuses of remote work, based on my experience so far.

                                                                                      + Flexible hours
                                                                                      – Potentially boundless work

                                                                                      By far, one of the top perks of remote work with a distributed team is truly flexible hours. I only have one or two synchronous meetings a week, and in the rest of my time I'm free to work the hours I prefer. No one expects me to be online at specific times, as long as the work gets done and I respond to pings within a reasonable time. As I'm a morning person, this means that I typically work a few hours in the early morning, take a long break (e.g., to surf or run some errands), and then work a few more hours in the afternoon or early evening.

                                                                                      The potential downside of such flexibility is not being able to stop working, especially as most of my colleagues are in Europe and North America. I deal with this by avoiding all work communications during my designated non-work hours. For example, I don't have any work-related apps on my phone, I keep all my work tabs in a separate tab group, and I turn Slack off when I'm not working. I found that this approach sets enough of a boundary between my work and personal life, though I do end up thinking about work problems outside work hours occasionally.

                                                                                      + More time for non-work activities
                                                                                      – There’s never enough time!

                                                                                      Not commuting freed up the equivalent of a workday in my schedule. In addition, having flexible hours means that I can make time in the middle of the day for leisure activities like surfing and diving. However, it's still a full-time job, so I'm not completely free to pursue non-work activities. It often feels like there isn't enough time in the day, as I can always think of more stuff I'd like to do. But my current situation is much better than having to commute on a daily basis. Even though it's been a relatively short time, I find the idea of going back to full-time office work hard to imagine.

                                                                                      + No need to attend an office
                                                                                      – Possible isolation from colleagues (and the real world)

                                                                                      Offices – especially open-plan offices – are not great places to get work done. This is definitely the case with work that requires a high level of concentration over uninterrupted blocks of time, like coding and data analysis. Working from home is great for avoiding distractions – there's no need for silly horse blinders here (though I do enjoy looking at the bird and lizard action outside my window).

                                                                                      One good thing about offices is the physical availability of colleagues. It's easy to ask others for feedback, socialise over drinks or shared meals, and keep up to date with company politics. Automattic works around the lack of daily physical interaction by running a few meetups a year. The number of people attending a meetup can vary from a handful for team meetups, to hundreds for the annual Grand Meetup. In all cases, the idea is to bring employees together for up to a week at a time to work and socialise. In my experience, the everyday distance creates a craving to attend meetups. I've never worked in a place where co-workers were so enthusiastic about spending so much time together – with non-distributed companies, team building is often seen as a chore. I suppose that the physical distance makes us appreciate the opportunity to be together and make the most of this precious time – it's a bit like being in a long-distance relationship.

                                                                                      That said, in the majority of the time, isolation can be a problem. As I'm based in Australia, I probably feel it more than others – most of my teammates are offline during my work hours, which means that there's no one to chat with on Slack. This isn't a huge issue, but I do need to ensure I get enough social interaction through other avenues. As the jobs page of Bandcamp (another distributed company) used to say: "If you do not have a strong social structure outside of work then employment at Bandcamp will likely lead to heart disease and an early death. We’re hiring!"

                                                                                      + Most communication is written
                                                                                      – Information overload

                                                                                      As Automattic is a fully-distributed company, most of the communication is done in writing. The main tools are Slack and internal forums called P2s (emails are rarely used). This makes catching up on the latest company news easy in comparison to places that rely more heavily on synchronous meetings. The downside of so much written communication is potential information overload. It is impossible to follow all the P2 posts, and even keeping up with stuff I should know can sometimes be overwhelming. I especially feel it in the mornings, as most of my colleagues work while I'm sleeping. Therefore, catching up on everything that happened overnight and responding to pings often takes over an hour – things are rarely as I left them when I last logged off. I experience this same feeling of being overwhelmed when coming back from vacation. Depending on the length of time away, it can take days to catch up. On the plus side, this process doesn't rely on someone filling me in – it's all there for me to read.

                                                                                      + Free trips around the world
                                                                                      – Jet lag and flying

                                                                                      As noted above, Automatticians meet in person a few times a year. Since joining, I attended meetups in Montreal, Whistler, Playa del Carmen, Bali, and Orlando. In some cases, I used the opportunity for personal trips near the meetup locations. Such trips can be a lot of fun. However, the obvious downside when travelling from Australia is that getting to meetups usually involves days of jetlag and long flights (e.g., the 17-hour Dallas to Sydney trip). Nonetheless, I still enjoy the travel opportunities. For example, I doubt I would have ever visited Florida and snorkelled with manatees if it wasn't for Automattic.

                                                                                      + Exposure to diverse opinions and people
                                                                                      – Cultural differences can pose challenges

                                                                                      Australia's population is made up of many migrants, especially in the tech industry. However, all such migrants have some familiarity with Australian culture and values. The composition of Automattic's workforce is even more diverse, and it lacks the unifying factor of everyone choosing to live in the same place. This is mostly positive, as I find the exposure to a diverse set of people interesting, and everyone tends to be friendly, welcoming, and focused on the work rather than on cultural differences. However, it's important to be aware of differences in communication styles. There's also a wider range of cultural sensitivities than when working with a more homogeneous group. Still, I haven't found it to be much of an issue, possibly because I'm already used to being a migrant. For example, moving to Australia from Israel required some adjustment of my communication style to be less direct.

                                                                                      Closing words

                                                                                      Overall, I like working with Automattic. For me, the positives outweigh the negatives, as evidenced by the fact that it’s the longest I’ve been in one position since 2012. Doing remote data science work doesn’t seem particularly different to doing any other sort of non-physical work remotely. I hope that more companies will join Automattic and the growing list of remote companies, and offer their employees the option to work from wherever they’re most productive.

                                                                                      Update (March 2019): I also covered similar topics in a Data Science Sydney talk about a day in the life of a remote data scientist.

                                                                                      Subscribe
                                                                                        -
                                                                                        Alternatively, subscribe to RSS feed.

                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                        parentscool
                                                                                        2018-11-04 01:37:44

                                                                                        I have been working remotely for WRI for nearly 2 years, and I can resonate with almost everything you have said. Great blog!

                                                                                        Triparna Ray
                                                                                        2018-11-05 05:53:51

                                                                                        Interested. Though not trained as Data scientist yet but as BI consultant with experience over a decade. Let me know if you have any opportunity.

                                                                                        Sreekanth Yasa
                                                                                        2018-11-06 02:29:18

                                                                                        I am working for Accenture as Analyst. The article is very similar to my real life. I pursued data science from top university and worked on few capstone projects.
                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by diff --git a/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/index.html b/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/index.html index 283cd513c..e4592210a 100644 --- a/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/index.html +++ b/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/index.html @@ -2,7 +2,7 @@

                                                                                        The most practical causal inference book I’ve read (is still a draft)

                                                                                        December 24, 2018

                                                                                        I’ve been interested in the area of causal inference in the past few years. In my opinion it’s more exciting and relevant to everyday life than more hyped data science areas like deep learning. However, I’ve found it hard to apply what I’ve learned about causal inference to my work. Now, I believe I’ve finally found a book with practical techniques that I can use on real problems: Causal Inference by Miguel Hernán and Jamie Robins. It is available for free from their site, but is still in draft mode. This post is a short summary of the reasons why I think Causal Inference is a great practical resource.

                                                                                        One of the things that sets Causal Inference apart from other books on the topic is the background of its authors. Hernán and Robins are both epidemiologists, which means they often have to deal with data with strong limitations on sample size and feasibility of experiments. Decisions driven by causal inference in epidemiology can often make the difference between life and death of individuals. Hence, the book is full of practical examples.

                                                                                        The book focuses on randomised controlled trials and well-defined interventions as the basis of causal inference from both experimental and observational data. As the authors show, even with randomised experiments, the analysis often requires using observational causal inference tools due to factors like selection and measurement biases. Their insistence on well-defined interventions is particularly refreshing, as one of the things that bothers me about the writings of Judea Pearl (a prominent researcher of causal inference) is the vagueness of statements like “smoking causes cancer” and “mud doesn’t cause rain”. The need for well-defined interventions was summarised by Hernán in the article Does water kill? A call for less casual causal inferences.

                                                                                        Unlike some other resources, Causal Inference doesn’t appear to be too dogmatic about the framework used for modelling causality. I’m not an expert on where each idea originated, but it seems like the authors mix elements from the potential outcomes framework and from Pearl’s graphical models. They also don’t neglect time as an important consideration in cause-and-effect relationships. In fact, the third part of the book is dedicated to the topic of time-varying treatments and effects.

                                                                                        The practicality of the book is also demonstrated by the fact that it comes with code examples in multiple languages. In addition, the authors don’t dwell too much on the philosophy of causality. While it is a fascinating topic, the opening paragraphs of the book make its goals clear:

                                                                                        By reading this book you are expressing an interest in learning about causal inference. But, as a human being, you have already mastered the fundamental concepts of causal inference. You certainly know what a causal effect is; you clearly understand the difference between association and causation; and you have used this knowledge constantly throughout your life. In fact, had you not understood these causal concepts, you would have not survived long enough to read this chapter–or even to learn to read. As a toddler you would have jumped right into the swimming pool after observing that those who did so were later able to reach the jam jar. As a teenager, you would have skied down the most dangerous slopes after observing that those who did so were more likely to win the next ski race. As a parent, you would have refused to give antibiotics to your sick child after observing that those children who took their medicines were less likely to be playing in the park the next day.

                                                                                        Since you already understand the definition of causal effect and the difference between association and causation, do not expect to gain deep conceptual insights from this chapter. Rather, the purpose of this chapter is to introduce mathematical notation that formalizes the causal intuition that you already possess. Make sure that you can match your causal intuition with the mathematical notation introduced here. This notation is necessary to precisely define causal concepts, and we will use it throughout the book.

                                                                                        I won’t try to summarise the technical aspects of the book – partly because I don’t fully understand it all, and partly because the book itself is already a summary of a very rich research area. However, I’m likely to go back and reread the book in the future, with the goal of applying the techniques from the book to my work. I’d also like to take Hernán’s causal inference course as a way of practising what I’ve learned from the book. For people who want a non-technical summary of the topics covered by the book, I recommend the article The c-word: Scientific euphemisms do not improve causal inference from observational data. If you’re curious about other (less practical) causality books I’ve read, check out my causal inference resource list and my two previous posts on the topic: Why you should stop worrying about deep learning and deepen your understanding of causality instead and Diving deeper into causality: Pearl, Kleinberg, Hill, and untested assumptions.

                                                                                        Subscribe
                                                                                          -
                                                                                          Alternatively, subscribe to RSS feed.

                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/index.html b/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/index.html index 932c22dc2..de0058695 100644 --- a/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/index.html +++ b/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/index.html @@ -13,7 +13,7 @@ https://yanirseroussi.com/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/dont-believe-everything-you-read-on-the-internet-lincoln.jpg 576w," src=https://yanirseroussi.com/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/dont-believe-everything-you-read-on-the-internet-lincoln.jpg alt="Abraham Lincoln internet quote" loading=lazy>

                                                                                          Update (Oct 2019): I published a post summarising a talk I gave on the topic, complete with simulation code that illustrates the issues with some bootstrapping algorithms.

                                                                                          Subscribe
                                                                                            -
                                                                                            Alternatively, subscribe to RSS feed.

                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                            Ralph Haygood
                                                                                            2019-01-08 00:09:43

                                                                                            “the basic bootstrap makes no assumption about the underlying distribution of the data”: I suppose bootstrapping per se doesn’t, but some things people like to use it for do. For example, suppose the original sample is from a Cauchy distribution, and bootstrapping is used to compute a confidence interval around the sample mean; no matter how many bootstrap replicates are used, the computed interval is worthless, because the original distribution doesn’t have a mean. Of course, that’s an extreme case unlikely to arise in practice, but it immediately raises doubt that guidelines like “n ≥ 101 for the bootstrap t” should be applied uncritically. As you obviously agree, it’s best to know and think about where the data came from when deciding which statistical methods to apply.

                                                                                            Ingo Rohlfing
                                                                                            2019-01-09 09:52:57

                                                                                            Learned a lot from the post. One question on the CIs for means and the difference between the means. If the two CIs for the means do not overlap, does it always imply that the difference is significant? Or can the error go in both ways, meaning that it is possible to have non-overlapping CIs and the CI of the difference includes 0?

                                                                                            Boris Gorelik
                                                                                            2019-01-15 12:38:17

                                                                                            Reblogged this on Boris Gorelik and commented:

                                                                                            Anything is better when bootstrapped. Read my co-worker’s post on bootstrapping. Also make sure following the links Yanir gives to support his claims

                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2019/10/06/bootstrapping-the-right-way/index.html b/2019/10/06/bootstrapping-the-right-way/index.html index 38c94581a..14d310379 100644 --- a/2019/10/06/bootstrapping-the-right-way/index.html +++ b/2019/10/06/bootstrapping-the-right-way/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2019/10/06/bootstrapping-the-right-way/revenue-confidence-intervals_hua7f3e259e998045c935f50c75e8eb77d_43359_1500x0_resize_box_3.png 1500w," src=https://yanirseroussi.com/2019/10/06/bootstrapping-the-right-way/revenue-confidence-intervals_hua7f3e259e998045c935f50c75e8eb77d_43359_800x0_resize_box_3.png alt="Revenue confidence intervals" loading=lazy>

                                                                                            The figure shows how the accuracy of confidence interval estimation varies by algorithm, sample size, and the number of bootstrapping resamples on a synthetic revenue dataset. This sort of dataset may occur in freemium scenarios, where several product variations are offered at a few price tiers, including a price of zero (i.e., free). In all cases, the dashed line denotes the requested confidence level of 95%, i.e., the true difference in means between the two revenue distributions should be inside the confidence interval in approximately 95% of the simulations for it to be accurate. Unfortunately, it is clear that both the percentile and BCa algorithms perform poorly on the simulated data. Even with a sample size of 10K, they both yield “95%” confidence intervals that contain the true difference in means less than 90% of the time, i.e., the intervals are too narrow. By contrast, the studentized algorithm gets much closer to the requested confidence level, but this comes at the price of considerably longer runtime due to the need for nested bootstrapping.

                                                                                            Note that the results presented in the talk are slightly different from the figure above. The difference is due to a small bug in the simulation code: I used a constant random seed for all the bootstrapping simulation iterations (every iteration still contained different data). This has led to the surprising finding that accuracy with 10,000 resamples was lower than with 1,000 resamples. I attributed that finding to dataset quirks, and noted that my results may not generalise to all cases. Indeed, I recently ran a similar set of experiments on different data as part of my work at Automattic, and found that the studentized algorithm accuracy wasn’t as impressive as the results shown here.

                                                                                            In addition to synthetic data, the experiments I ran at Automattic included an implementation of an idea by my colleague, Demet Dagdelen: Test accuracy on samples from the full population for a given period (e.g., all sales over a calendar year). In such cases, the full population is well-defined. Therefore, we know the value of the “true” parameters, and we can run the same simulations as on synthetic data. While I can’t share that data, I can say that all algorithms performed much worse on real data than on simulated data. Therefore, we decided to follow the penultimate takeaway and use a parametric Bayesian approach for modelling our data. We may share insights from that line of work on data.blog in the future. In the meantime, comments are very welcome!

                                                                                            Update: You can find more accurate simulations in this post.

                                                                                            Subscribe
                                                                                              -
                                                                                              Alternatively, subscribe to RSS feed.

                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                              Boris Gorelik
                                                                                              2019-10-06 07:54:37

                                                                                              Reblogged this on Boris Gorelik and commented:

                                                                                              Many years ago, I terribly overfit a model which caused losses of a lot of shekels (a LOT). It’s not that I wasn’t aware of the potential overfitting. I was. Among other things, I used several bootstrapping simulations. It turns out that I applied the bootstrapping in a wrong way. My particular problem was that I “forgot” about confounding parameters and that I “forgot” that peeping into the future is a bad thing.

                                                                                              Anyhow, Yanir Seroussi, my coworker data scientist, gave a very good talk on bootstrapping.

                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/index.html b/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/index.html index af49ad81b..c5c0c742b 100644 --- a/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/index.html +++ b/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/index.html @@ -2,7 +2,7 @@

                                                                                              A day in the life of a remote data scientist

                                                                                              December 11, 2019

                                                                                              Earlier this year, I gave a talk titled A Day in the Life of a Remote Data Scientist at the Data Science Sydney meetup. The talk covered similar ground to a post I published on remote data science work, with additional details on my daily schedule and projects, some gifs and Sydney jokes, heckling by the audience, and a Q&A session. I managed to watch it a few months ago without cringing too much, so it’s about time to post it here. The slides are on my GitHub, as is my list of established remote companies, which you may find useful if you want to join the remote work fun.

                                                                                              Subscribe
                                                                                                -
                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2020/01/11/software-commodities-are-eating-interesting-data-science-work/index.html b/2020/01/11/software-commodities-are-eating-interesting-data-science-work/index.html index 33bd3ecaa..0b7852aae 100644 --- a/2020/01/11/software-commodities-are-eating-interesting-data-science-work/index.html +++ b/2020/01/11/software-commodities-are-eating-interesting-data-science-work/index.html @@ -11,7 +11,7 @@ https://yanirseroussi.com/2020/01/11/software-commodities-are-eating-interesting-data-science-work/science-versus-engineering.png 581w," src=https://yanirseroussi.com/2020/01/11/software-commodities-are-eating-interesting-data-science-work/science-versus-engineering.png alt="Information flow in science and engineering" loading=lazy>

                                                                                                Many problems in data “science” are actually engineering problems – described best by the flow on the right (source)

                                                                                                Some of my first jobs as a data scientist in industry involved building recommender systems. With recommender systems, much of the work is on the system around the recommendation algorithm. That is, building a recommender system was always mostly an engineering problem. However, these days we have services like AWS Personalize, which does most of the heavy lifting around recommendation. This makes the deployment of recommender systems a pure engineering problem. Like many other problems, recommender systems have been commodified.

                                                                                                I have not done much with deep learning, but there the general trend is even more apparent: Useful innovations quickly turn into tools. Examples include library evolution from Theano to TensorFlow, and commodified prediction services from companies like Google, Amazon, and Microsoft. If you want to use a deep learning service in your application, you probably don’t need a data scientist or even a machine learning engineer. A solid software engineer who can pick the right tools should be enough.

                                                                                                How to remain relevant?

                                                                                                So where does this leave us? It seems to be a more general phenomenon. Essentially every problem that requires specialised knowledge and is valuable ends up attracting repeatable solutions that obviate the need for deep thinking and manual work. These solutions are software commodities. Deploying them is a matter of writing some glue code and fitting them into the overall system – an engineering problem. Implementing data science components to compete with commodities may be interesting and fun, but it’s usually a waste of time when there’s a generic solution that is good enough.

                                                                                                As an individual data scientist, what can you do when your speciality becomes a software commodity? I see a few options:

                                                                                                1. Embrace the engineering angle. Become good (or better) at engineering solutions. Be pragmatic. Do what it takes to get the job done. This is probably easier for data scientists like me, who have an engineering background, than for more research/analysis-oriented data scientists. Such data scientists sometimes sneer at engineering work, claiming it’s “fake” data science. Fake or not, solid engineering tools can easily make stubborn data scientists obsolete.
                                                                                                2. Keep building custom solutions even when viable commodities exist. While this may be more fun for the individual, I believe it isn’t a sustainable approach. The cost of building and maintaining custom solutions will typically be higher than the cost of commodity solutions. Insisting on custom solutions seems like a recipe for becoming irrelevant.
                                                                                                3. Keep adapting and moving to non-commodity areas. Some things are easier to automate than others. For example, building a machine learning pipeline when the problem is well-defined is relatively easy, but deciding what features to create typically requires some domain expertise. In addition, new research keeps coming out in areas that are less hot than machine learning. One such area is causal inference, where there are still solutions that are yet to be commodified.
                                                                                                4. Move to the cutting edge. If you want to research novel methods, a “standard” data scientist position may not be for you. Many industry positions are focused on applying proven solutions to a specific organisation. If that doesn’t sound like fun, you’re better off moving to academia or joining a commercial research group.

                                                                                                Are there any other options I don’t see? Let me know in the comments!

                                                                                                Subscribe
                                                                                                  -
                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                  John Chew
                                                                                                  2020-01-11 09:35:15

                                                                                                  This is the same conclusion I reached when deciding between deepening data science skills vs engineering; now I’m deeper into cloud services and off-the-shelf ML tools.

                                                                                                  Yanir Seroussi
                                                                                                  2020-01-12 21:20:19

                                                                                                  Good points, thanks Boris!

                                                                                                  antonisbtr
                                                                                                  2020-01-16 18:09:52

                                                                                                  Hi Yanir!

                                                                                                  The post really reasonated with me. I find more and more that I do engineering during my day than science.

                                                                                                  I believe the data engineering part, including cloud and full stack development skills, will prove to be the skills that keep you relevant in industry. If you combine these with knowledge on which techniques to use regarding data science and machine learning, then you can be unstoppable.

                                                                                                  Otherwise, as you said, it’s better to stay in academia.

                                                                                                  Best, Antonios

                                                                                                  Audrey D
                                                                                                  2021-07-14 18:24:40

                                                                                                  Hi Yanir,

                                                                                                  I am glad I found your post. I am switching careers and want to work with data for social good. I learned data analysis, and am thinking of explore machine learning, see if that’s something for me. Being an engineer (not tech related), I would definitely be content with your option 1, where I understand what’s going on behind the scenes but don’t get into research and maths.

                                                                                                  Interesting to know what the trend is in the field.

                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | diff --git a/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/index.html b/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/index.html index fe35f59cd..8a89268e9 100644 --- a/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/index.html +++ b/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/index.html @@ -44,7 +44,7 @@ https://yanirseroussi.com/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/revenue-confidence-intervals-with-bounds_hu2767cc513317d344ae2cdd478297ecf3_56731_1500x0_resize_box_3.png 1500w," src=https://yanirseroussi.com/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/revenue-confidence-intervals-with-bounds_hu2767cc513317d344ae2cdd478297ecf3_56731_800x0_resize_box_3.png alt="Revenue confidence intervals with bounds" loading=lazy>

                                                                                                  Notes: See this notebook for code – use the same environment as the original notebook. The cover photo is by Dima D from Pexels.

                                                                                                  Subscribe
                                                                                                    -
                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2021/04/05/some-highlights-from-2020/index.html b/2021/04/05/some-highlights-from-2020/index.html index 800fcc328..0b2cf2662 100644 --- a/2021/04/05/some-highlights-from-2020/index.html +++ b/2021/04/05/some-highlights-from-2020/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2021/04/05/some-highlights-from-2020/bougainville-reef-wall-dive_hufa91eac262d7ccfc888de175482140e1_5626271_1500x0_resize_q75_box.jpg 1500w," src=https://yanirseroussi.com/2021/04/05/some-highlights-from-2020/bougainville-reef-wall-dive_hufa91eac262d7ccfc888de175482140e1_5626271_800x0_resize_q75_box.jpg alt="Bougainville reef wall dive" loading=lazy>

                                                                                                    While data from RLS dives helps global conservation efforts, diving also reminds me that there’s still so much left to save and conserve

                                                                                                    Reef Life Survey (RLS). Another distributed organisation that I’m involved with, and a worthwhile cause, is the RLS foundation. I previously posted about my experiences with RLS offline data collection and visualisation of the collected data, and have since helped with quite a few RLS surveys. Despite lockdowns and border closures, 2020 was no exception: I participated in the Lord Howe biennial surveys in February (just before the initial lockdown), and was fortunate to join a survey trip from Airlie Beach to Thursday Island in October (long after lockdown lifted in the lucky state of Queensland). I also joined the 38(!) author list of Establishing the ecological basis for conservation of shallow marine life using Reef Life Survey – a Biological Conservation journal paper covering RLS’s history, methodology, outcomes, and more. Finally, I was surprised and honoured to receive the Scoresby Shepherd Award for doing the most RLS surveys in the 2019-20 financial year. It was clearly a bit of a slow year due to the pandemic, but it’s always nice to get recognised. Overall, 2020 was definitely a good year for my participation in RLS and I’m planning on contributing more in 2021, especially with help around organising and conducting surveys in Southeast Queensland.

                                                                                                    Technical work. My main “day job” focus in 2020 was on being the tech lead for Automattic’s new experimentation platform (ExPlat). This aligns well with my long-standing interest in causal inference. Among other things, it gave me an opportunity to apply my favourite approach to Bayesian A/B testing in the wild, and get excited about other interesting causal inference work we have in the pipeline. Now that ExPlat’s foundation is mostly in place, we are planning on sharing much of our work on data.blog. My colleague Aaron just published the first post in the series, and my post on ExPlat’s architecture will be next. Subscribe to data.blog to get updates!

                                                                                                    Subscribe
                                                                                                      -
                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2021/10/07/my-work-with-automattic/index.html b/2021/10/07/my-work-with-automattic/index.html index 2815a0a51..93c5760ac 100644 --- a/2021/10/07/my-work-with-automattic/index.html +++ b/2021/10/07/my-work-with-automattic/index.html @@ -2,7 +2,7 @@

                                                                                                      My work with Automattic

                                                                                                      October 7, 2021
                                                                                                      Bing thinks I looked like this while working at Automattic.

                                                                                                      Bing thinks I looked like this while working at Automattic.

                                                                                                      Automattic is the company behind WordPress.com, Tumblr, Jetpack, WooCommerce, and several other products. I worked with Automattic as a Type B Data Scientist (i.e., I mostly built and deployed code to production) from May 2017 to October 2021. This post is back-dated to my last day with the company to make it fit nicely into my post timeline, but I’m actually writing this in July 2023. The magic of time travel! 🪄

                                                                                                      A nice perk of working with Automattic was getting to write about my work on company blogs. When my website was on WordPress.com, I used the reblogging feature to share those posts here, but they never looked great. One of the first projects I completed after leaving Automattic was migrating my site from WordPress.com to Hugo, which made the reblog posts look even worse. Now all those reblogs redirect here, thanks to Hugo’s aliases feature.

                                                                                                      Anyway, here are some highlights from my Automattic work along with links to the relevant posts:

                                                                                                      On this website, you can also read about how I ended up joining Automattic and on some of the reasons behind my decision to leave the company.

                                                                                                      Subscribe
                                                                                                        -
                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/index.html b/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/index.html index 0c99d87fe..1369c9384 100644 --- a/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/index.html +++ b/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/index.html @@ -8,7 +8,7 @@ 100vw" srcset="https://yanirseroussi.com/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/before-and-after.png 640w," src=https://yanirseroussi.com/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/before-and-after.png alt="Before and after look of a recent post" loading=lazy>

                                                                                                        Before and after look of a recent post

                                                                                                        As I was eager to finish the initial migration, I avoided spending too much time on non-critical tasks. These include following all the SEO best practices, increasing page speed, applying various style tweaks, and other small changes. With more control over my site, I now have the power to incrementally address such tasks over time.

                                                                                                        In summary, I found the migration rewarding and educational. It was also fun to go through old posts and get motivated to publish more frequently. I’m looking forward to shifting my focus to the content – stay tuned for new posts!

                                                                                                        Subscribe
                                                                                                          -
                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/index.html b/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/index.html index b40dbefd3..116ac01a5 100644 --- a/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/index.html +++ b/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/dog-philosophy.png 1024w," src=https://yanirseroussi.com/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/dog-philosophy_huc3cf4931803219559fd375fea6b748c2_70831_800x0_resize_box_3.png alt="Dog Philosophy by Three Panel Soul" loading=lazy>

                                                                                                          Source: Three Panel Soul - dog philosophy

                                                                                                          Of course, what constitutes good is an open question, which I touched on in the talk. Other key points include:

                                                                                                          • The modelling context is much broader than any machine learning model. Considering context is where human brains shine.
                                                                                                          • Thoughtlessness can have a negative impact on society and on your career.
                                                                                                          • Moral values vary across time, space, cultures, and individuals, e.g., along five moral foundations.
                                                                                                          • Any data scientist, machine learning engineer, or modern human should develop their critical thinking skills. The Calling Bullshit course from the University of Washington is a great starting point – essentially Data Literacy 101.
                                                                                                          • Bullshit is easier to detect than call. Deciding on a level of bullshit calling is like tuning a model’s learning rate.

                                                                                                          A good chunk of the talk was spent on the case study on criminal machine learning from the Calling Bullshit website. I was pleased with the level of engagement on this segment, especially since a lockdown forced us to deliver the class online at short notice. You can watch the full talk below (my part ends after 24 minutes), view the slides here, and check out supplementary materials from all mentors on GitHub.

                                                                                                          Subscribe
                                                                                                            -
                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2022/01/14/analysis-strategies-in-online-a-b-experiments/index.html b/2022/01/14/analysis-strategies-in-online-a-b-experiments/index.html index 74b18971d..1a5ce0575 100644 --- a/2022/01/14/analysis-strategies-in-online-a-b-experiments/index.html +++ b/2022/01/14/analysis-strategies-in-online-a-b-experiments/index.html @@ -4,7 +4,7 @@ sign up today!

                                                                                                            A simplified mockup of the variants. Which one would you choose?

                                                                                                            Placing this scenario into the above diagram, if we were to simply change the text, i.e., apply the Treatment to everyone, we wouldn’t be able to confidently tell whether the text change was the cause of any observed difference in the conversion rate. For example, if our release coincided with a surge of interest in cryptocurrency, this surge may be one of the Unknowns that would cause more motivated users to come to our exchange and sign up. That is, the surge would affect both exposure to the Treatment and the Outcome.

                                                                                                            When we run an ideal A/B experiment, we don’t have this problem. Factors like a surge of interest in crypto don’t affect the assignment of users to the control group A (“sign up”) and the treatment group B (“sign up today!”). We can compare the conversion rates across the groups, estimate random variability with our favourite A/B testing calculator, and rejoice. Right?

                                                                                                            Well, not so fast…

                                                                                                            Problems, problems…

                                                                                                            In the ideal scenario, all the users that were assigned to one of the experiment groups experience their assigned variant and produce a measurable outcome. In our running example, the groups are A: control and B: treatment with a simple exposure of seeing “sign up” for the former and “sign up today!” for the latter. The outcome is a successful signup or an absence of a signup. To make the outcome well-defined, it’s often a good idea to limit outcome measurement to events that happen (or don’t happen) within a reasonable attribution window from exposure or assignment. In our example, a reasonable attribution window is probably on the order of hours, as we don’t expect the call-to-action text to have long-lasting effects.

                                                                                                            Potential deviations from the ideal scenario include:

                                                                                                            • Assignment of ineligible users. In our running example, these may be bots or users that already have an account. If we include many ineligible users in our analysis, we may underestimate the effect size even if their distribution across groups is uniform.
                                                                                                            • Crossovers. These are users that manage to experience both variants. For example, they may come across our site on mobile with the “sign up today!” text, and then switch to desktop and see the “sign up” message. Depending on the instrumentation we have in place, we may not be able to detect such users, or we may only detect them if they sign up on one device and then log in on the other device.
                                                                                                            • Assignment without exposure. Due to implementation constraints, we may not be guaranteed that assigned users are actually exposed to the treatment and control. In our running example, it may be that the assignment is done on the backend while exposure happens conditionally and asynchronously on the frontend – some users may bounce in the gap between assignment and exposure, and never see the call-to-action text.
                                                                                                            • Multiple exposures. Once a user has been assigned, they may get exposed to the treatment and control multiple times (without crossing over). In our example, they may visit the landing page repeatedly and see the “sign up” or “sign up today!” text multiple times before deciding to sign up.

                                                                                                            Epidemiologist jargon and analysis strategies

                                                                                                            While clinical trials are more tightly controlled than online A/B experiments, they are also susceptible to problems like assignment of ineligible patients and non-adherence to treatment (e.g., crossover, non-exposure, and multiple exposures). Hence, much has been written on addressing these problems at the analysis stage. However, when researching the topic, overcoming the domain-specific language barrier was a bit of a challenge, as the terminology used by online experimenters is different from the terminology used by epidemiologists. Fortunately, I came across the term intention-to-treat at some point, which opened the door to decades of research on the topic.

                                                                                                            Two papers I found useful are Intention-to-treat concept: A review (Gupta, 2011) and Guidelines for estimating causal effects in pragmatic randomized trials (Murray, Swanson, and Hernán, 2019). Seeing Miguel Hernán on the author list was an especially positive signal for me, as he is responsible for some of my favourite resources on causal inference, including the most practical book I’ve read on the topic.

                                                                                                            The definitions and guidelines from these two papers provide a solid foundation for thinking about problems of ineligibility and non-adherence. Specifically, Gupta defines intention-to-treat as an analysis strategy “that includes all randomized patients in the groups to which they were randomly assigned, regardless of their adherence with the entry criteria, regardless of the treatment they actually received, and regardless of subsequent withdrawal from treatment or deviation from the protocol.”

                                                                                                            There are often good reasons to exclude some randomised participants from analysis. Depending on the exclusions, this may or may not bias the results. The use of conservative exclusions can be described as modified intention-to-treat, which according to Gupta “allows the exclusion of some randomized subjects in a justified way (such as patients who were deemed ineligible after randomization or certain patients who never started treatment). However, the definition given to the modified ITT (mITT) in randomized controlled trials has been found to be irregular and arbitrary because there is a lack of consistent guidelines for its application. The mITT analysis allows a subjective approach in entry criteria, which may lead to confusion, inaccurate results and bias.”

                                                                                                            Exclusions and further adjustments are usually an attempt to estimate the per-protocol effect, which is defined by Murray, Swanson, and Hernán as “the effect of receiving the assigned treatment strategies throughout the follow-up as specified in the study protocol.” Unfortunately, obtaining a valid estimate of the per-protocol effect isn’t trivial: “To validly estimate the per-protocol effect, baseline variables which predict adherence and are prognostic for the outcome need to be accounted for, either through direct adjustment or via an instrumental variable analysis. Yet two commonly used analytic approaches do not incorporate any such adjustment: (1) Naïve per-protocol analysis, that is, restricting the analytic subset to adherent individuals; and (2) As-treated analysis, that is, comparing individuals based on the treatment they choose.” In other words, if we’re not careful, the per-protocol analysis may become analogous to an uncontrolled experiment, as depicted at the top of the diagram above.

                                                                                                            What should be done in practice?

                                                                                                            From my reading of the clinical trial literature, the tendency is to use multiple analysis strategies. For example, the first guideline noted by Murray, Swanson, and Hernán is: “To adequately guide decision making by all stakeholders, report estimates of both the intention-to-treat effect and the per-protocol effect, as well as methods and key conditions underlying the estimation procedures.” This echoes the 1988 US FDA guidelines that require applicants to provide an intention-to-treat analysis in addition to the applicant’s preferred per-protocol analyses. Similarly, the 1998 European Medicines Agency guidelines provide more details on the intention-to-treat, modified intention-to-treat, and per-protocol strategies, stating that: “In general, it is advantageous to demonstrate a lack of sensitivity of the principal trial results to alternative choices of the set of subjects analysed. […] When the full analysis set and the per protocol set lead to essentially the same conclusions, confidence in the trial results is increased, bearing in mind, however, that the need to exclude a substantial proportion of subjects from the per protocol analysis throws some doubt on the overall validity of the trial.”

                                                                                                            While the stakes in online experiments are typically much lower than in human drug approval, I believe that applying multiple analysis strategies is still a great idea. We did that for Automattic’s experimentation platform, where we flagged discrepancies between the strategies if they led to conflicting conclusions. One downside of this approach is that it complicates the presentation of results in comparison to using a single strategy. If you face the same challenge, you may draw inspiration from seeing how it’s addressed by the open source frontend of Automattic’s experimentation platform.

                                                                                                            Going back to our running example, we can perform the following analyses to deal with the deviations noted above:

                                                                                                            • Intention-to-treat. Includes all users based on their initial group assignment, regardless of what variant they were exposed to.
                                                                                                            • Modified intention-to-treat: No ineligible users. This applies to cases where we detect the ineligibility after assignment, but the eligibility criteria are based on factors that could have been known before the experiment. Hence, it should be safe to exclude the ineligible users after the fact. In our example, excluding bots and existing users should increase the observed effect size, but not change the preferred variant.
                                                                                                            • Modified intention-to-treat: No crossovers. If we have a mechanism to detect some crossovers, excluding them and comparing the results to the intention-to-treat analysis may uncover implementation bugs. It’s worth noting that crossovers shouldn’t occur in cases where we can uniquely identify users at all stages of the experiment – it is a problem that is more likely to occur when dealing with anonymous users, as in our landing page example. As such, and given the inability to detect all crossovers, A/B experiments should be avoided when users are highly motivated to cross over. For example, displaying different price levels based on anonymous and transient identifiers like cookies is often a bad idea.
                                                                                                            • Naive per-protocol: Exposed users. For this analysis, we’d only include users that were exposed to the control and treatment texts. As noted by Murray, Swanson, and Hernán, this is naive because we should adjust our estimates based on variables that predict exposure. However, if missing exposures are only due to the inherent limitations of online experiments, this falls more under the modified intention-to-treat criterion noted by Gupta, of excluding “patients who never started treatment”. Things get more complicated if we wish to use each exposure as a distinct starting point for measuring multiple assignment windows (the multiple exposures scenario above), which is akin to patients choosing their own dosage – far from a controlled experiment. For automated analysis, it’s better to use the first exposure as the attribution window start, as it should be unaffected by the experiment variants.

                                                                                                            For all analysis approaches, it’s critical to verify that there is no sample ratio mismatch in the analysed population, i.e., that the distribution of users across variants matches what we expect from a random assignment. If this isn’t the case, manual analysis by a qualified data scientist is needed. The result of this manual analysis may be that the results should be discarded, as sample ratio mismatches are a common indicator of implementation bugs. This is discussed in detail in the book Trustworthy Online Controlled Experiments, which also includes a chapter on exposure-based analysis (called triggering in the book). Among other recommendations, the authors suggest analysing the unexposed users. If everything goes as expected, metrics for the assigned-but-unexposed populations would behave like A/A experiment metrics, i.e., any differences between the groups should be due to random variability.

                                                                                                            Having rigorous consistency checks in place and falling back to manual analysis when any discrepancies are detected should help avoid the pitfalls of unsafe user exclusions that’d bias the results. Given the need for careful adjustments to get a valid per-protocol estimate in case anything goes wrong, it is often best to fix any underlying issues and rerun the experiment. Usually, this is much cheaper to do in an online setting than in clinical trials.

                                                                                                            Closing thoughts and further reading

                                                                                                            Once you move from the theory of experimentation to the practice of running experiments in the real world, you discover the many complexities involved in doing it well. This applies whether you’re an epidemiologist or an online experimenter. As noted in the preface to the trustworthy experiments book: “Getting numbers is easy; getting numbers you can trust is hard!”

                                                                                                            This post only scratched the surface of one area of experimentation: Deciding what population to analyse once the experiment was run. There is, of course, a lot more to online experimentation and causal inference than what I could cover here. But I hope that this message is clear: Approach experimentation with humility, and aim to learn from a broad set of teachers rather than limit yourself to the relatively-recent developments in online experiments.

                                                                                                            As mentioned above, some resources that are worth reading to learn more include my favourite causal inference book, the trustworthy experiments book, and the guidelines for pragmatic trials. There are also a bunch of resources on my causal inference list, and my post on Bayesian A/B testing should be of interest if you made it to this point. Finally, I’m always happy to discuss these topics, so feel free to contact me or leave a comment with your thoughts.

                                                                                                            Cover image by Tumisu from Pixabay

                                                                                                            Subscribe
                                                                                                              -
                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/index.html b/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/index.html index 38cda9ec6..929d92e7d 100644 --- a/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/index.html +++ b/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/index.html @@ -6,7 +6,7 @@ https://yanirseroussi.com/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/real-programmers-xkcd.png 740w," src=https://yanirseroussi.com/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/real-programmers-xkcd.png alt="xkcd: Real Programmers" loading=lazy>

                                                                                                              Real Programmers don’t do easy things. Source: xkcd.

                                                                                                              That said, the fast.ai library isn’t perfect. Debugging can be a bit frustrating, as it tries to do a lot of things automatically and mutates many objects in surprising ways. Its documentation is also somewhat lacking (perhaps due to the use of notebooks as the primary development environment), and its naming conventions can be a bit odd (especially the overuse of acronyms). But these are minor annoyances rather than blockers. It does work well for its main use cases, and it’s possible to go down to the PyTorch level when necessary.

                                                                                                              5. As always, it’s all about the data and how you use it. This is hardly a new lesson for me, but it’s worth reiterating. Given the maturity of computer vision and other machine learning packages, data scientists should focus on getting relevant data and understanding the problem well. As Andrej Karpathy noted in his 2019 recipe for training neural nets, and I said in my 2014 Kaggle tips, you should aim to become one with the data.

                                                                                                              6. FOMO will always be there, but it can be lessened. In general, I care more about making useful things than about using the latest techniques. This is why I prioritised working with RLS to get my tool deployed. Still, FOMO in data science is a well-documented phenomenon, and I suffer from it too. It’s encouraging that – given some free time and a clear head – it’s not that hard to catch up on recent developments. This is made especially easy by the availability of many free resources, like fast.ai. The main thing to remember is to focus on principles rather than worry about the million methods and tools that are out there – it was true in 1911, and it’s still true today.

                                                                                                              Subscribe
                                                                                                                -
                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                Ishan Nangia
                                                                                                                2023-06-14 10:51:18

                                                                                                                Loved this short blog. Planning to transition to Climate tech as a DS guy and am slowly cultivating a pent-up passion for conserving the marine life so too many things that I can relate to here haha

                                                                                                                Yanir Seroussi
                                                                                                                2023-06-15 21:46:03

                                                                                                                Thank you! Good luck with the transition. 🙂
                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/index.html b/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/index.html index de1812e05..25e67ebee 100644 --- a/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/index.html +++ b/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/index.html @@ -11,7 +11,7 @@ https://yanirseroussi.com/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/rectangular-nightmare.png 1158w," src=https://yanirseroussi.com/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/rectangular-nightmare_hua7b55b1a9144008aca94017bec6c6b89_1030812_800x0_resize_box_3.png alt="Tweet by Jesse Zook Mann: The jury is still out of what is causing depression to surge in the past decade. Showing a meme that says that 'after a nice long day of typing stuff into my nightmare rectangle at work, I like to come home and stare at my portable nightmare rectangle'." loading=lazy>

                                                                                                                Rectangles are useful, but we also need time without them.

                                                                                                                1. In addition to the mission, Automattic CEO Matt Mullenweg has shared his vision of making Automattic the Berkshire Hathaway of the internet, a goal that I find even less inspiring. ↩︎

                                                                                                                2. While I was aware of the mission when I joined Automattic in 2017, it wasn’t a critical criterion for me. Over the years, I’ve become more conscious of the role online platforms play in destabilising societies. I now believe that it’s important for platforms to acknowledge their responsibilities and delegate power to external regulators, e.g., as Facebook is doing with their Oversight Board (which is still an imperfect solution). ↩︎

                                                                                                                3. It’s also an open question whether it’s possible to offset things like the harmful work of the Murdoch press. ↩︎

                                                                                                                4. I still like the 2018 definition, so hopefully I’m done with defining data science. ↩︎

                                                                                                                5. According to a recent study, data science is seen as an incredibly boring job. Not sexy at all. ↩︎

                                                                                                                6. Despite this, I wasn’t particularly looking forward to going back to frequent long-haul flights – it was an aspect of Automattic work I never liked. This made the prospect of post-pandemic work with Automattic less appealing, even without considering the climate impact of so much flying. ↩︎

                                                                                                                Subscribe
                                                                                                                  -
                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                  Jessie Ross
                                                                                                                  2022-06-06 07:55:44

                                                                                                                  This is a very well written post and it’s great to hear your reasoning on this. Congrats on your new position!
                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2022/09/12/causal-machine-learning-book-draft-review/index.html b/2022/09/12/causal-machine-learning-book-draft-review/index.html index 9b7d5caa5..63d86387d 100644 --- a/2022/09/12/causal-machine-learning-book-draft-review/index.html +++ b/2022/09/12/causal-machine-learning-book-draft-review/index.html @@ -8,7 +8,7 @@ Source: Wikipedia – ABO blood group system (retrieved on 2022-09-11)." loading=lazy>

                                                                                                                  When observing parent phenotypes (ABO blood types) without genotypes, grandparent phenotypes are informative.
                                                                                                                  Source: Wikipedia – ABO blood group system (retrieved on 2022-09-11).

                                                                                                                  I also struggle with overly-casual statements like this one:

                                                                                                                  Suppose we were interested in modeling the relationship between altitude and temperature. The two are clearly correlated; the higher up you go, the colder it gets. However, you know temperature doesn’t cause altitude, otherwise heating the air within a city would cause the city to fly. Altitude is the cause, and temperature is the effect.

                                                                                                                  In fact, heating the air within a city would cause the heated air to rise. And extremely high heat can melt a city and the land it’s on, thereby causing a reduction in its altitude.

                                                                                                                  While this may seem like nitpicking, ill-defined causal graphs are a serious problem. One of my favourite papers on the topic is Does water kill? A call for less casual causal inferences, which argues that "[while] it is impossible to provide an absolutely precise definition of a version of treatment […] specification of versions of treatment is required only until no meaningful vagueness remains". However, “declaring a version of treatment sufficiently well-defined is a matter of agreement among experts based on the available substantive knowledge” because we don’t have an objective way of determining that treatments are well-defined. In line with this thinking, the book may benefit from reducing the variety of examples in favour of a handful of small datasets that are more well-defined and defensible.

                                                                                                                  Despite these shortcomings, I found chapters 1-3 of Causal Machine Learning pleasant enough to get through, and I look forward to reading more. Getting into DoWhy and other related packages has been on my list, and I’m sure I’ll learn a lot by following the MEAP. After tracking the field for almost a decade and complaining about the relative hype levels of deep learning and causal inference, it’s great to see a practical book that aims to marry the two. The Causal Revolution is truly upon us.

                                                                                                                  1. It is almost inevitable that when pointing out the mistakes of others I will make mistakes myself. I apologise for any mistakes and welcome feedback. ↩︎

                                                                                                                  Subscribe
                                                                                                                    -
                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2022/12/11/chatgpt-is-transformative-ai/index.html b/2022/12/11/chatgpt-is-transformative-ai/index.html index 1ba385823..e19e2bf17 100644 --- a/2022/12/11/chatgpt-is-transformative-ai/index.html +++ b/2022/12/11/chatgpt-is-transformative-ai/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2022/12/11/chatgpt-is-transformative-ai/chat-gpt-is-transformative-ai.png 1450w," src=https://yanirseroussi.com/2022/12/11/chatgpt-is-transformative-ai/chat-gpt-is-transformative-ai_hu2a6554fee21e5adc8b4e5b4193592a03_183217_800x0_resize_box_3.png alt="ChatGPT Prompt: 'Write a paragraph on the topic "ChatGPT is transformative AI".' Response: 'ChatGPT is a transformative AI that has the potential to change the way we interact with technology. It is a powerful language model that can understand and generate human-like text, allowing it to engage in natural language conversations with people. This technology has the potential to revolutionize many fields, including customer service, education, and even personal communication. With ChatGPT, we can have more natural and intuitive interactions with AI, making it easier for us to access information and get things done. In short, ChatGPT represents a major advancement in AI technology and has the potential to transform how we live and work.'" loading=lazy>

                                                                                                                    ChatGPT in its own words

                                                                                                                    Subscribe
                                                                                                                      -
                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                      Ralph Haygood
                                                                                                                      2022-12-12 23:32:24

                                                                                                                      I’m amused that the first sample on the home page features the user asking, “this code is not working like i expect - how do i fix it?” In my mind, I hear the voice of HAL (Douglas Rain) answer, “This sort of thing has cropped up before, and it has always been due to human error.”

                                                                                                                      I’ve been assuming ChatGPT is just the latest specimen of what typically passes for AI these days, a system with an elaborate model of utterances disconnected from any deeper and richer model of the world to which utterances refer, hence brittle and shallow. (Such systems more or less realize John Searle’s “Chinese room” scenario, although unlike Searle, I don’t think they represent any fundamental limit on AI, merely the current, crude state of the art.) However, you’ve convinced me to try it out.

                                                                                                                      Yanir Seroussi
                                                                                                                      2022-12-14 19:58:34

                                                                                                                      Thanks Ralph! It’s definitely still early days, but it feels like a step change in chatbot tech, much like the significant improvements in image recognition from a decade ago. I can see it only getting more capable with all the interaction data that they’re collecting.
                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/04/21/remaining-relevant-as-a-small-language-model/index.html b/2023/04/21/remaining-relevant-as-a-small-language-model/index.html index 39c342902..c877d6b4f 100644 --- a/2023/04/21/remaining-relevant-as-a-small-language-model/index.html +++ b/2023/04/21/remaining-relevant-as-a-small-language-model/index.html @@ -11,7 +11,7 @@ https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a-small-language-model/getting-replaced-by-ai-using-ai.jpg 500w," src=https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a-small-language-model/getting-replaced-by-ai-using-ai.jpg alt="Image of people using AI getting replaced by AI using AI" loading=lazy>

                                                                                                                      What might happen once you’ve finally mastered the latest AI tools. Source: Reddit

                                                                                                                      Subscribe
                                                                                                                        -
                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                        JCHEW
                                                                                                                        2023-04-21 08:44:02

                                                                                                                        Reading of the despair of the many silent/invisible contributors of intellectual property copied, without financial recompense, which went into these LLMs. It’s a training data gold rush/free for all right now, just as unedifying and thoughtless as actual gold rushes in history were. I fear this will also quickly lead to Apple-style closed gardens of proprietary creative talent (think of top artists, writers and thinkers signed up to train AI rather than creating content for direct human consumption). Counter-ML tech like Glaze will only delay the inevitable. https://glaze.cs.uchicago.edu/.

                                                                                                                        I can think of some ways where governments might respond, e.g. special taxes and incentives on AI businesses to fund creative academies and collectives, much as public universities are today.

                                                                                                                        Yanir Seroussi
                                                                                                                        2023-04-22 00:34:00

                                                                                                                        Thanks John! Yeah, I doubt that tech like Glaze can be made future proof, as they admit on that page. Besides, I lean more towards the view that all creative work is derivative and copying isn’t theft. Copyright mostly protects platforms and businesses rather than individuals. While I empathise with individuals who feel like their work is being exploited without their permission, I don’t see the training of machine learning models as being that different from artists learning from other artists.

                                                                                                                        Thoughtful government intervention would be great, but it’s unlikely to be applied in a timely manner or evenly across jurisdictions.

                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/05/26/how-hackable-are-automated-coding-assessments/index.html b/2023/05/26/how-hackable-are-automated-coding-assessments/index.html index 9e6b6c1d6..497974e35 100644 --- a/2023/05/26/how-hackable-are-automated-coding-assessments/index.html +++ b/2023/05/26/how-hackable-are-automated-coding-assessments/index.html @@ -17,7 +17,7 @@ https://yanirseroussi.com/2023/05/26/how-hackable-are-automated-coding-assessments/blind-code-assessment-game.png 1244w," src=https://yanirseroussi.com/2023/05/26/how-hackable-are-automated-coding-assessments/blind-code-assessment-game_hu4596bb32dd7a8ed7b3602d876d56970a_249362_800x0_resize_box_3.png alt="Screenshot from a Blind thread on coding assessments" loading=lazy>

                                                                                                                        Some comments from a Blind thread on coding assessments. Seeing it all as a somewhat-useful game is probably the way to go.

                                                                                                                        Note: I reached out to CodeSignal for a comment on this post, but haven’t heard back after more than a week.

                                                                                                                        1. As with many Paul Graham essays, I find myself in agreement with some of his ideas and disagreement with others. But hackable tests are definitely a thing, e.g., see teaching to the test and Campbell’s law. ↩︎

                                                                                                                        2. Taking a machine learning analogy, asking the same questions repeatedly is likely to lead to overfitting. Drawing new questions from the same distribution is akin to adding a validation set, while dealing with the sort of problems encountered outside standardised tests is indicative of the generalisation error of the test taker. ↩︎

                                                                                                                        3. It says a lot about the hackability of higher education that the Australian government requires a PhD graduate from a top Australian university to prove that their English skills haven’t deteriorated after four years in Australia. Similarly, companies that look at educational pedigree but put recent graduates through their own set of tests implicitly distrust the grades given by universities. ↩︎

                                                                                                                        4. The only time you’re likely to face ridiculous time pressures that are measured in minutes is when something breaks in production. Production issues can be minimised through investment in solid processes and quality over a project’s lifetime. That is, you go slow to go fast and avoid fire-fighting. Take-home exams and real-work simulations are more reflective of the sort of thinking that’s required from senior engineers because good ideas often manifest when you take the time to design a system and avoid jumping into code-like-hell mode. Going with the first thing that comes to mind is a habit that’s better left to chatbots. ↩︎

                                                                                                                        5. Preparation is partly a function of motivation to pass the test, which is a positive indicator despite being unrelated to possessing the skills the test purports to measure. In my case, motivation to maximise the score was lacking, so the company got useful information out of my imperfect score. Why was my motivation lacking? Because the role seemed interesting enough to apply to, but not worth working too hard to get. The opportunity cost of neglecting my other endeavours in favour of test hacking seemed too high. ↩︎

                                                                                                                        6. See page 11 of the linked survey. Like other materials from CodeSignal, it’s somewhat comical. They state that “candidates view CodeSignal assessments more favorably than timed coding assessments in general (p = 0.034)”, but looking at the table, the mean score given to CodeSignal assessments is 3.41 / 5, while general timed coding assessments were given a mean score of 3.37. That is, a difference of 0.04 – it’s hard to call this practically significant, despite the p-value. Could it be that CodeSignal’s IO psychologists missed the many memos on p-value pitfalls, such as the one by the American Statistical Association? In any case, if they consider the 0.04 difference to be notable, why do they say nothing about the 0.06 difference in favour of take-home coding assignments or the 0.17 difference in favour of coding interviews? Personally, I’d also report the full distribution rather than just the means. It’s easy enough to visualise a five-point scale. ↩︎

                                                                                                                        Subscribe
                                                                                                                          -
                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/index.html b/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/index.html index a8cfa01c2..5f396dd18 100644 --- a/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/index.html +++ b/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/index.html @@ -11,7 +11,7 @@ https://yanirseroussi.com/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/dilbert-big-data.jpg 904w," src=https://yanirseroussi.com/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/dilbert-big-data_hu52b7900d669ddfe03bdd894fda67e436_330047_800x0_resize_q75_box.jpg alt="Dilbert big data" loading=lazy>

                                                                                                                          Will big data ever make a resurgence as a silver bullet?

                                                                                                                          (M5) Lack of automated source-code control. While McConnell’s 2008 report found this to be a low-frequency mistake, data science may have helped resuscitate it. This is due to multiple factors:

                                                                                                                          • Using notebooks for development and experimentation. I use notebooks myself – they are popular for a reason. However, they don’t play well with source control systems without additional tooling. For example, it’s hard to collaborate on notebooks as people do on plain code – just try merging notebook changes from multiple authors for a bit of fun.
                                                                                                                          • Many data scientists came from fields where source control wasn’t common. This is probably decreasing now with Git being the standard source control system, but it wasn’t the case about a decade ago.
                                                                                                                          • Data transformations don’t always live under source control. For example, analytics flows might be buried in stored procedures or database views, or worse – copied around by analysts. Again, this is changing thanks to a growing awareness and the rise of tools like dbt, but not everyone has adopted the modern data stack yet.

                                                                                                                          Related to a lack of source-code control is a lack of control over model versioning and data lineage, as it takes some experience to develop an appreciation of the need for versioning and reproducibility. Still, it’s easy to end up with a big mess even with an awareness of these issues and the best intentions. It’s hard to control everything.

                                                                                                                          Learning from history while moving forward

                                                                                                                          Data science is maturing. We’ve gone from a “sexy” field to people calling themselves “recovering data scientists”, saying goodbye to the field, and declaring that “there will be no data science job titles by 2029”. Anecdotally, it seems to me like data science is becoming less of a failure mode of software engineering – perhaps because data scientists are no longer expected to single-handedly ship complex software systems.

                                                                                                                          Personally, I still struggle with giving a concise title to what I do, just like in 2012. Data scientist has become a loaded term – some see data science as a cost centre that fails to deliver tangible results. As I’ve never stopped doing software engineering, I try to emphasise it by saying that I’m a full-stack data scientist and software engineer. This is a bit of a mouthful, and full-stack is also a loaded term, but it seems apt because I’ve shipped production code that ranges from old-school C on pre-Android phones through data pipelines to web applications. My main concern these days is putting my skills to good use, especially within climate tech and related areas. But as I’m freelancing, I went with Data & AI Consultant as my LinkedIn job title – maybe my PhD specialisation in AI wasn’t so silly after all…?

                                                                                                                          Subscribe
                                                                                                                            -
                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/index.html b/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/index.html index 823b0146b..fd05fd742 100644 --- a/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/index.html +++ b/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/ditcherville-39-i-love-your-email-list.png 1352w," src=https://yanirseroussi.com/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/ditcherville-39-i-love-your-email-list_hue4c7fc22fce0574ea830c25d91fbdb2a_253127_800x0_resize_box_3.png alt="Comic strip showing trendy alternatives to email, year after year" loading=lazy>

                                                                                                                            Source: Ditcherville: I love your email list! CC BY-NC 4.0 by Jonathan Stark

                                                                                                                            Subscribe
                                                                                                                              -
                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/10/25/lessons-from-reluctant-data-engineering/index.html b/2023/10/25/lessons-from-reluctant-data-engineering/index.html index 28e0ebf52..ddc6d4caf 100644 --- a/2023/10/25/lessons-from-reluctant-data-engineering/index.html +++ b/2023/10/25/lessons-from-reluctant-data-engineering/index.html @@ -2,7 +2,7 @@

                                                                                                                              Lessons from reluctant data engineering

                                                                                                                              October 25, 2023
                                                                                                                              Yanir Seroussi presenting at DataEngBytes Brisbane 2023

                                                                                                                              In May 2023, I submitted the following talk abstract to the Brisbane DataEngBytes conference.

                                                                                                                              As we all know, solid data engineering is essential to the success of data science and AI applications. And yet, people often get excited about fancy machine learning models and neglect the data engineering layer. This is totally understandable: playing with data in a throwaway notebook is more relaxing than dealing with a data pipeline that keeps finding ways to break in production.

                                                                                                                              In this talk, I’ll share lessons on data engineering from a data science perspective. Everywhere I’ve worked, from small start-ups to established companies, I’ve found that I had to do some data engineering if I wanted my work to ever get to production. While I’ve always been reluctant to do too much of it, my engineering background has placed me in a better position to do it than colleagues who started off as analysts and academics.

                                                                                                                              You could call my work full-stack data science, reluctant data engineering, or some other data & AI thing. Whatever it is, I hope that my talk will help us all play better with each other, across all layers of the data stack.

                                                                                                                              As I don’t identify as a data engineer and have never attended a DataEngBytes conference, I didn’t know whether my talk would fit the agenda. However, it seemed harmless to submit an abstract and see how it goes.

                                                                                                                              When I got the acceptance notification and realised I had to turn my abstract into a coherent talk, I was a bit wary of lacking a good grasp of who’s in my audience. However, when the full agenda was published, I realised that the focus of the conference won’t be on arcane data engineering knowledge, given that one of the keynotes was titled “How The Full-Stack Data Scientist Is STILL The Sexiest Job”. It turned out that despite the name and tagline (“by data engineers, for data engineers”), DataEngBytes was a great event for all data professionals.

                                                                                                                              Here’s the video of the talk (slides):


                                                                                                                              Quick summary. I start off with a disclaimer, stating that I am not a data engineer. Then I show evidence that the market values data engineering more than data science, given the ratio of Data Engineer to Data Scientist job ads (x3 in the AU$100-150k compensation range; x4 in the AU$200k+ range).1 I follow that observation with another disclaimer, stating that some of my lessons may be obvious or better learnt the hard way (as I often have to learn and relearn lessons). Then I detail five chronologically ordered snippets and their corresponding lessons:

                                                                                                                              1. 2012: My first data science job, where we made mistakes around technology choice and premature optimisation. The lesson is that shiny tech ain’t always shiny. Like all lessons, this one ends with a quote that shows that what I learned wasn’t entirely new. The first quote is by Donald Knuth from 1974: “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”
                                                                                                                              2. 2013: My first head of data science job, where we solved real scaling issues by following principles and adapting solutions to our situation. The lesson is that shiny tech can be transformative; but principles beat tools, which goes with a 1911 quote by Harrington Emerson: “As to methods, there may be a million and then some, but principles are few. The person who grasps principles can successfully select their own methods. The person who tries methods, ignoring principles, is sure to have trouble.”2
                                                                                                                              3. 2015: My first enterprise consulting stint, where I experienced being a not-so-useful data scientist and working with some not-so-useful data engineers. This led me to dabble in “shadow IT” (a term I learned at the conference), and build a separate Python machine learning pipeline to work around various limitations. The lesson is that you should solve problems; don’t be the problem, or in the words of circa 2004 Google: “Focus on the user and all else will follow.”
                                                                                                                              4. 2017: My first remote data science job, where I played around with many job functions across the data stack and went down various data rabbit holes. The lesson is to go deep; trust but verify, which goes with a 1999 quote by Eric S. Raymond: “Given enough eyeballs, all bugs are shallow.”
                                                                                                                              5. 2022: My first committed climate and biodiversity moves (still a work in progress). The lesson is that tech & titles are tools; focus on what matters, but recall Rabbi Tarfon’s quote from almost two thousand years ago: “You are not obliged to complete the work, but neither are you free to desist from it.”

                                                                                                                              The main takeaway from the talk is that data problems have human roots – and human solutions. This is because:

                                                                                                                              • Humans get excited by shiny tech… and produce transformative tech.
                                                                                                                              • Humans optimise prematurely… and when it makes sense.
                                                                                                                              • Humans can act as unreasonable blockers… and as the users we serve.
                                                                                                                              • Humans generate messy data… and clean it up.
                                                                                                                              • Humans get distracted by tools… and use them for beneficial ends.

                                                                                                                              1. This is based on Seek searches for jobs advertised in July 2023. Given the limitations of Seek search, it’s not an accurate representation of the demand for each role, as the results included all ads that mentioned the terms. One could also argue that data engineers tend to change jobs more than data scientists, fuelling demand. Despite this, I think the results support the general message around the value of data engineering, especially as others have noted the need for 4-5 data engineers per data scientist in organisations with complex data engineering requirements. ↩︎

                                                                                                                              2. Emerson referred to man rather than person in the original quote, but I took the liberty to make it gender-neutral and retain the original message. ↩︎

                                                                                                                              Subscribe
                                                                                                                                -
                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/index.html b/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/index.html index 43f5816a8..fe121601a 100644 --- a/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/index.html +++ b/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/index.html @@ -2,7 +2,7 @@

                                                                                                                                Supporting volunteer monitoring of marine biodiversity with modern web and data tools

                                                                                                                                November 29, 2023
                                                                                                                                Screenshot of Reef Species of the World

                                                                                                                                I’ve been volunteering with the Reef Life Survey (RLS) citizen science project since 2015. RLS volunteers follow the same underwater visual census methodology that has been in use for decades, thereby producing data series that help inform the management of marine ecosystems. In simpler terms, we count fish (and some invertebrates), and this helps various organisations know what’s happening underwater. Among other places, RLS data has been used in scientific publications in Nature and elsewhere, and to inform the management of Australian marine parks.

                                                                                                                                Over the years, I created a few online tools to help volunteers with survey work. These included web apps to visualise survey results and study species, as well as infer species from underwater photos. More recently, I agreed to help with the general maintenance of the non-WordPress parts of the RLS website and backend (somewhat reluctantly, but I suppose that’s what happens when you do things out of love).

                                                                                                                                Taking greater responsibility to help the tech side of RLS along with an alignment of the research grant stars led to an opportunity to revamp the Reef Species of the World (RSoW) section – a collection of over 5,000 species with in-situ photos, descriptions, and empirical distributions derived from RLS surveys. My focus in this project was on product management, data pipelines, and backend work. I was joined by my brother, Uri Seroussi, who was in charge of front-end development (which became much more substantial than in the original RSoW).

                                                                                                                                The original RSoW was a traditional PHP application that relied on a MySQL database to serve requests, with most of the HTML constructed on the server. By contrast, we re-architected the new RSoW as a progressive web app using Next.js, which has the following advantages and new features:

                                                                                                                                • Fully static site: served faster with reduced server load.
                                                                                                                                • Faster search and navigation: happens on the front-end without round-trips to the server.
                                                                                                                                • Installable app with offline availability: RSoW can now be installed as a mobile or desktop app, and run without an internet connection.
                                                                                                                                • Client-side image classification: offline availability includes image classification in the browser, which is useful when surveying in remote areas.
                                                                                                                                • Replacement of previous tools and pipelines: providing a more consistent user experience and improved data reliability.

                                                                                                                                The rest of this post provides details on the architecture and implementation of the new RSoW and its underlying data and machine learning pipelines. But the best way of getting a feel for the data and the tools is to have a play yourself.

                                                                                                                                The new RSoW architecture diagram reflects the compromises between rebuilding and retaining legacy systems

                                                                                                                                The new RSoW architecture diagram reflects the compromises between rebuilding and retaining legacy systems

                                                                                                                                The RSoW web app

                                                                                                                                We didn’t start with a blank slate: RSoW was already a public website, with many individual species pages ranking well on web searches (the main source of traffic). As such, a guiding principle was to retain as much of the original functionality as possible, and then build new features on top of it.

                                                                                                                                When approaching a legacy codebase, there’s always the question of whether rebuilding parts or all of it is a worthwhile endeavour. As Jason Cohen notes, a more apt name for “legacy code” is “revenue code”, i.e., the code that embodies all the original and changed requirements, and has withstood the test of time. Even though RLS’s code isn’t meant to generate revenue, it’s always easy to mess things up when re-implementing existing functionality.

                                                                                                                                The main reasons we decided on a rewrite of the front-end were:

                                                                                                                                • User experience: Speed things up, as some species searches were pretty slow due to server round-trips and inefficient database queries.
                                                                                                                                • Extensibility: Make it easier to add new features.
                                                                                                                                • Offline availability: This is impossible with a traditional PHP back-end, but feasible if all the data and code gets shipped to the client.

                                                                                                                                We chose Next.js as the front-end framework since it’s well-established and supports static exports. Parts of the RLS website run on WordPress, so it’s easy to add statically-generated pages and serve them efficiently via Cloudflare (I wasn’t keen on complicating the stack by adding a Node backend). With static exports, we regenerate all the species pages whenever the data changes, which means that end-user page requests don’t need to touch the database. In addition, the main search page downloads three JSONs with all the data it needs to perform any species search (see sites.json, species.json, and surveys.json in the rls-data repo). Minified and compressed, these JSONs add up to less than 2MB of data, which isn’t tiny, but it is a small price to pay to avoid hitting the database. The JSONs also cache well on Cloudflare, like the rest of the web app’s files.

                                                                                                                                From a user perspective, replicating the original functionality was the less exciting part of the project. Faster and less buggy code is obviously better, but once feature parity was achieved, we turned our attention to some new features:

                                                                                                                                • Supporting offline availability and installation by turning RSoW into a progressive web app: On its face, this was supposed to be simple given the next-pwa package, but it turned out to be a bit tricky because the original package was abandoned, and due to multiple layers of caching. It’s well-known that cache invalidation is one of the two hard problems in computer science (along with naming things and off-by-one errors), and progressive web apps offer a lovely variety of caches to deal with – everything needs to be cached on the client for offline availability. We got there after some tinkering and dealing with head-scratching bugs, some of which were caused by other caching layers in addition to the client-side caches (including Cloudflare and some misconfiguration of an early version of the app).
                                                                                                                                • Knowledge test: A separate grant came along and Uri had the opportunity to extend RSoW by adding a section that helps test new volunteers ahead of them joining RLS.
                                                                                                                                • Species frequency exploration: Bringing in the full functionality from the first tool I built for RLS back in 2017.
                                                                                                                                • Client-side image classification: Deprecating the Streamlit app I built a couple of years ago.

                                                                                                                                Data and machine learning pipelines

                                                                                                                                On the back-end, there was an opportunity to simplify things by retiring the original PHP code that processed survey data in favour of the pipelines I implemented in the rls-data repo. Ultimately, survey data comes from the Australian Ocean Data Network (AODN), which holds many more datasets in addition to RLS. Originally, the PHP code that processed survey data into the MySQL database evolved separately from rls-data, which I implemented to generate JSONs for the tools I built. As rls-data is an open source project and the raw survey data is relatively small (<1GB), it made sense to process it with a daily GitHub Action (GHA) script that runs for free. The resultant JSONs are committed to the repo, which means that any unexpected changes are easily tracked (I keep an eye on the commits). It was simple to expand the existing rls-data pipelines to generate all the JSONs needed to serve RSoW, and then say goodbye to the PHP code that implemented similar functionality.

                                                                                                                                I’m aware that running data pipelines with GitHub Actions isn’t going to win any awards for sophistication, but it’s a great fit for this project. The key principle is to use the right tool for the job, not the shiniest tool.

                                                                                                                                One part of the original RSoW that we barely touched was the management interface, which allows RLS admins to update species data and upload pictures. The gains from replacing the admin part of RSoW would have been negligible, so it still runs the old PHP code on top of MySQL. Unfortunately, this meant I couldn’t retire all the PHP data pipelines, as species data also comes from the Australian Ocean Data Network and is joined with the edits made by RLS admins. This exemplifies the pragmatism that one often needs to apply when faced with legacy revenue systems: If a system works and there’s no real benefit to replacing it, sticking with the old system is the right thing to do (even if it makes your architecture diagram more complicated).

                                                                                                                                I have big plans to improve the machine learning model for inferring RLS species from user images, but it’s somehow never a priority. For RSoW, I did make it a priority to support serving the model with a simple API, but then I decided it’d be worth the effort to export it to ONNX for client-side image classification. This was partly driven by curiosity about ONNX, but it also had two key benefits: (1) support for offline classification; and (2) simplified & cheaper serving architecture, as ONNX models can be served from S3 and don’t require RLS to pay for server-side compute.

                                                                                                                                As to the machine learning pipelines, they all need to be manually triggered, which is fine since the image data changes slowly. These pipelines are implemented in notebooks and the command-line interface of the ichthywhat repo. I have a bit of a dream of this being an early precursor to complete automation of RLS data collection, with the historical RLS data series continued by divers who would mostly serve as video takers and fish scarers (using cameras without human divers would lead to different biases in the data). However, this is a big project that is probably best left to my next PhD, i.e., it may never happen.

                                                                                                                                In the meantime, I hope to continue diving with RLS, and aim to make pragmatic decisions to keep RSoW running and supporting the community.

                                                                                                                                Subscribe
                                                                                                                                  -
                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/index.html b/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/index.html index 278914df0..3defbc6be 100644 --- a/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/index.html +++ b/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/index.html @@ -2,7 +2,7 @@

                                                                                                                                  New decade, new tagline: Data & AI for Impact

                                                                                                                                  January 19, 2024
                                                                                                                                  Logo of Yanir Seroussi's consulting services, depicting a wave and an up-and-to-the-right graph.

                                                                                                                                  Exactly a decade ago, on 19th January 2014, I published my first post on this website (Kaggle beginner tips). In most of the following years, my tagline was Data Science and Beyond. While the beyond bit gave me an excuse to write about various topics, most posts were indeed around data science – an area that also became broader (arguably to the point of uselessness).

                                                                                                                                  While I’ve never abandoned my software engineering roots, the broadening of data science means that many data scientists can no longer be assumed to possess solid engineering skills. Therefore, I changed the tagline last year to Engineering Data Science & More. However, this didn’t feel quite right – some people now have an adverse reaction to any mention of data science, after negative experiences of failed projects.

                                                                                                                                  Recently, I switched the tagline to be both broader and narrower: Data & AI for Nature. However, upon reflection and given some feedback, I realised that the Nature bit may be off-putting to some people who do impactful work in the space but have different motivations. Therefore, I decided to go with Data & AI for Impact (for now…).

                                                                                                                                  More importantly, I’m planning to revitalise my approach to publishing and audience engagement:

                                                                                                                                  • Post more frequently – aiming for weekly from February onwards.
                                                                                                                                  • Use the mailing list to email full posts, and as a two-way avenue for comments and conversations (as opposed to public comments, which are now closed).
                                                                                                                                  • Still publish both technical and high-level posts on Data & AI.
                                                                                                                                  • Produce content that’s specifically useful for startups and scaleups that are early on their Data & AI journey.
                                                                                                                                  • Showcase positive-impact applications of Data & AI tech – especially by startups in the climate and nature-positive space.

                                                                                                                                  With more frequent posts, what I publish should be quicker to produce and consume. This means I may lean more heavily on showcasing other people’s work – possibly through interviews. Other than that, here are some rough post ideas for the immediate future:

                                                                                                                                  • Series on a minimum viable data stack
                                                                                                                                  • Best practices and opinions on a startup’s first data hire
                                                                                                                                  • Answering questions people ask on the future of data science
                                                                                                                                  • My experience as a Data Tech Lead with Work on Climate
                                                                                                                                  • Use cases for ChatGPT and other LLMs
                                                                                                                                  • Catching up on different aspects of LLMs / AI tech
                                                                                                                                  • Opportunities for Data & AI professionals in the energy transition

                                                                                                                                  Historically, for each post I’ve published, about 5-10 ideas went unpublished. I hope that by aiming for shorter and lower-friction publishing, more posts will see the light of day.

                                                                                                                                  My long-term aims are to learn by publishing, apply my Data & AI skills towards more positive impact, and help others in the space. Rather than sinking into doom and gloom, I’d like to focus on positive applications of Data & AI tech that make our world better (in the spirit of publications like Volts).

                                                                                                                                  Call to action:

                                                                                                                                  • If this all sounds uninteresting to you, you’re welcome to unsubscribe – no hard feelings.
                                                                                                                                  • If you know people I should talk to and feature in future posts, I’d appreciate an intro.
                                                                                                                                  • If you have any suggestions, please send them by replying to any of my emails, or contact me through other means – I’d love to hear from you.
                                                                                                                                  Subscribe
                                                                                                                                    -
                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/index.html b/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/index.html index 9206b5665..a4603bda0 100644 --- a/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/index.html +++ b/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/index.html @@ -2,7 +2,7 @@

                                                                                                                                    Substance over titles: Your first data hire may be a data scientist

                                                                                                                                    February 5, 2024
                                                                                                                                    ChatGPT's version of a versatile data person

                                                                                                                                    If you search the web for ‘first startup data hire’, you may come across some strongly-worded advice claiming that this person must not be a data scientist, or that they must be a data engineer / analyst. In my view, being so prescriptive about titles risks missing out on great candidates. The reality is that titles in the data world are messy and fluid – it’s best to start by getting clear on what the data person is going to do, and proceed from there.

                                                                                                                                    Beyond titles, this post summarises my perspective on questions that arise around the first data hire, and presents some pointers to help you hire successfully.

                                                                                                                                    Assumptions and Timing

                                                                                                                                    Key assumption: If your startup needs a data hire, you’re probably at a stage where you’re starting to be limited by visibility into your data. You are generating revenue, but your data is all over the place (spreadsheets, dashboards of various tools, unstructured logs, etc.). There are important questions about your business that you can’t answer because you’re either not collecting the data, or because it’s too hard to gather it into a coherent story. No one on your technical team has informed opinions on tools to pick out of the dozens of options for warehousing / ingestion / transformation / analytics / orchestration.

                                                                                                                                    If this is the case, then data and machine learning isn’t core to your product. You need someone to set up your data pipelines and analytics. These will primarily serve internal-facing use cases, like driving marketing decisions. However, the first question to ask is: Do you really need to hire someone for a permanent full-time position?

                                                                                                                                    Personally, I’m biased in favour of not hiring (yet): You can get started on your data journey with a contractor or a part-time (aka fractional) person. This should give you a better understanding of your data needs, and get you to a better place in terms of data infrastructure and dashboards. This person may also want to become a full-timer down the track, or help you with hiring other data people.

                                                                                                                                    Remember that – by definition – premature hiring unnecessarily shortens your runway. Hiring and onboarding a full-timer would usually take longer than bringing on an experienced contractor. And if you need to let them go, it may adversely affect team morale. This doesn’t apply to contractors, who are expected to leave when their contract is over.

                                                                                                                                    That said, there is value in retaining a long-term owner of your data and analytics. Every business, dataset, and data stack have their quirks, so the familiarity that comes with long-term ownership is a definite point in favour of hiring for a permanent role. That said, you should still be open to part-time if you don’t have full-time needs yet.

                                                                                                                                    Titles and Skills

                                                                                                                                    If you do decide to hire for a permanent role, there are three other articles worth reading:

                                                                                                                                    • Andrew Bartholomew covers assumptions (similar to the above), responsibilities, skills, management, and the thorny question of titles. He says that the person’s title is “the least important question […] you’re hiring a Senior Analytics Engineer or a Senior Data Analyst, but in practice this person might prefer a Senior Data Scientist title, or Analytics Lead, or something else.” I agree with this and pretty much everything else in Andrew’s article, though it is important to align on the expectations implied by titles (more on this below).
                                                                                                                                    • Colleen Tartow advocates for hiring a senior data engineer. While Colleen’s advice is sensible, I’d be careful with following it blindly due to the messiness of titles and experiences. For example, you probably don’t want a data engineer who’s only worked with big companies, as there’s a risk that they’d over-engineer your data stack (initially, you’re aiming for a minimum viable data stack). Also, if they’ve only ever worn the data engineer hat, they may find it hard to uncover and communicate the insights you’re after.
                                                                                                                                    • Sebastian Hewing goes deep into the question of timing the hire as a function of product-market fit. I agree with most points, but disagree with this phrasing: “The last person you want is a Data Scientist. […] What you need, in my opinion, is a Head of Data & Analytics.” I believe that someone who has full-stack data science experience may make a great Head of Data & Analytics – it all comes down to skills and experiences rather than past titles, which can only ever tell a part of the story. That said, Sebastian does list a bunch of other data titles that the startup shouldn’t hire, so we probably agree on the essence of the role and the person. I especially like Sebastian’s emphasis on seeking a hands-on data person who can turn data into insights AND insights into action.

                                                                                                                                    As you can see, the three articles disagree on the question of titles, with Andrew’s being the most pragmatic. If you want to get even more confused, ask ChatGPT to summarise the collective wisdom of the internet: When I asked it “what should a startup’s first data hire be?”, ChatGPT suggested seven(!) roles with an “it depends” reason for each one. Personally, I’d go for a senior data generalist with an engineering background, who is also attentive to the business side. It’s highly doubtful you’d find someone who goes by this title, so you’ll need to figure out how to find and attract them. This is hard if you’re not familiar with the data space. It’s worth seeking help from data folks in your network, or starting with a contractor to bootstrap the process.

                                                                                                                                    Summary

                                                                                                                                    Putting it all together, once you’ve read the above articles, my opinion is that you should:

                                                                                                                                    1. Get clear on the business needs that’d be addressed by a data person.
                                                                                                                                    2. Err on the side of not hiring prematurely – consider a contractor or rely on your current employees.
                                                                                                                                    3. When you’re ready to hire, sketch out a high-level plan for the person’s first 90-180-360 days.
                                                                                                                                    4. Run the plan and job description by some data people you trust.
                                                                                                                                      • Possible title for the job ad: Data & Analytics Lead or Head of Data & Analytics (but you want a hands-on person, so make it clear that this is an individual contributor role initially).
                                                                                                                                      • Make the plan a part of the job ad – it helps with aligning expectations.
                                                                                                                                    5. Ideally, get data people you trust to help you with the hiring process.
                                                                                                                                    6. Screen out specialists early, regardless of past titles and pedigree.
                                                                                                                                    7. Make expectations as clear as possible during the hiring process – especially if the person hasn’t worked with a startup before.
                                                                                                                                    8. Hire someone who’s a great fit who would help take your business to the next level.

                                                                                                                                    Any thoughts or suggestions? Please contact me – I will make edits to this post based on feedback.

                                                                                                                                    Subscribe
                                                                                                                                      -
                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/index.html b/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/index.html index 998962897..69b1585b2 100644 --- a/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/index.html +++ b/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/index.html @@ -15,7 +15,7 @@ https://yanirseroussi.com/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/cheatsheet-final-pdf-attempt.webp 826w," src=https://yanirseroussi.com/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/cheatsheet-final-pdf-attempt_hua0a7c998a0b13e6d00449e04939ae63c_77530_800x0_resize_q75_h2_box_2.webp alt="ChatGPT's final cheatsheet attempt (screenshot of the PDF)" loading=lazy>

                                                                                                                                      Unfortunately, the cheatsheet partly lost touch with the book summary, but the bits ChatGPT decided to add aren’t terrible. Also, the drive was over, so it was time to bring the cheatsheet game to an end.

                                                                                                                                      Conclusion

                                                                                                                                      While this may seem like a pointless exercise, I’m pleased with these outcomes:

                                                                                                                                      • Relearning that getting ChatGPT to elaborate on summary bullet points can be useful – or at least somewhat entertaining.
                                                                                                                                      • Coming up with the HTML to PDF export path as a way to get ChatGPT to produce nice-looking PDFs.
                                                                                                                                      • Producing a lovely cheatsheet that contains some sound advice.

                                                                                                                                      Overall, I still find ChatGPT mind-blowing. My usage of it reduced a bit last year, but since I got the Plus subscription, it’s a completely different story. And the really amazing thing is that it’s still early days for this technology. Exciting times!

                                                                                                                                      Subscribe
                                                                                                                                        -
                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/02/19/building-your-startups-minimum-viable-data-stack/index.html b/2024/02/19/building-your-startups-minimum-viable-data-stack/index.html index 75d4c4ad7..5051caf17 100644 --- a/2024/02/19/building-your-startups-minimum-viable-data-stack/index.html +++ b/2024/02/19/building-your-startups-minimum-viable-data-stack/index.html @@ -2,7 +2,7 @@

                                                                                                                                        Building your startup's minimum viable data stack

                                                                                                                                        February 19, 2024
                                                                                                                                        Henrik Kniberg's drawing of a minimum viable product, showing the wrong way of doing it (non-functional iterations) and the right way of doing it (functional iterations).

                                                                                                                                        Minimum viability is about rapidly delivering incremental value. Source: Henrik Kniberg.

                                                                                                                                        In my post on your startup’s first data hire, I noted in passing that the hire’s initial role would be setting up the company’s minimum viable data stack. But what exactly is that?

                                                                                                                                        Conceptually, a minimum viable data stack follows the same principles as a minimum viable product. Breaking it up, it is:

                                                                                                                                        • Minimal: You don’t want to over-build beyond your resources. Instead, set up the simplest stack that satisfies the startup’s near-term needs. Then iterate based on feedback and new requirements.
                                                                                                                                        • Viable: While this is sometimes forgotten in favour of an over-emphasis on minimality, your data stack has to be viable. That is, it has to satisfy stakeholder needs through every iteration, as shown by the classic drawing above.
                                                                                                                                        • Data Stack: This is the product that’s getting shipped and built iteratively, consisting of the components listed below. As in my previous post, I’m assuming a startup where the data stack initially serves internal stakeholders. The main difference between a consumer-facing product and an internal-facing data stack is that the latter has users you can easily talk to and fewer unknowns. This makes the task of satisfying user needs easier.

                                                                                                                                        This post is the first in a series that will serve as a quick reference for those embarking on the journey of setting up a minimum viable data stack. Future posts will go deeper into each of the key components. However, I can only cover limited ground – readers are encouraged to consult books such as Fundamentals of Data Engineering for a more thorough treatment of the topics covered in the series.

                                                                                                                                        Components of a minimum viable data stack

                                                                                                                                        As we’re talking about a minimum viable data stack, this list of components isn’t exhaustive. The components I consider to be the bare minimum are:

                                                                                                                                        1. Storage: Where the data lives.
                                                                                                                                        2. Ingestion: How the data makes it into storage.
                                                                                                                                        3. Transformation: The layer that joins and changes raw data into more useful form.
                                                                                                                                        4. Analytics: Presentation layer, which can be consumed by non-technical stakeholders.

                                                                                                                                        Components that are perhaps conspicuous in their absence are: machine learning / AI (higher on data’s hierarchy of needs – I assume this will come later), data serving / querying (implicitly included in other layers), and orchestration (also implicit – I assume dependencies are initially simple enough so that any orchestration approach would work).

                                                                                                                                        Considerations beyond stack components

                                                                                                                                        There are at least two critical items to think of early on. I consider a data stack to be nonviable if no thought is given to:

                                                                                                                                        • Security. Ignore data security at your own peril. Examples abound of serious data breaches, which are often the result of trivial mistakes or poor design decisions. Starting off by implementing security checklists and following best practices like the principle of least privilege is way easier than trying to enforce them later on. A good technical read on the topic is Building Secure and Reliable Systems, though you’d need to cherry-pick principles to match your needs (it’s a book by Google, and your startup is not Google).
                                                                                                                                        • Privacy. You should be aware of compliance requirements for the data you store, especially when it comes to private and sensitive data. As with security, it’s much easier to start by complying with privacy requirements than retrofitting a stack once stakeholders have come to depend on data that shouldn’t be collected or retained. In the spirit of minimality, it’s best to err on the side of not storing sensitive data when it isn’t required. This also helps minimise the potential effect of breaches.

                                                                                                                                        Other key considerations include:

                                                                                                                                        • Data generation. I assume that the business is generating data from multiple sources, which need to be ingested into a single storage system to ultimately drive decisions. If there’s only one source system, it may be too early for a data hire, or for a more sophisticated data stack. When considering data sources, it’s important to keep in mind the three Vs of data: Volume, Velocity, and Variety.
                                                                                                                                        • Quality assurance. It’s important to have some automated checks in place to avoid breaking pipelines and maintain high data quality before making changes to production systems (e.g., changing transformation or ingestion code). Again, it’s easier to start with high standards for quality than enforce them retrospectively. Low quality is likely to result in low trust of any data or insights, making the stack nonviable.
                                                                                                                                        • Observability/monitoring/incident response. Inevitably, things will break in production. With good observability and incident response practices, the data team will proactively address such issues – ideally before any stakeholders notice. This goes hand in hand with setting high quality standards – the sort of culture that is easier to set early on than change down the track.
                                                                                                                                        • Data management. There are many items that fall under data management (e.g., see the list on Wikipedia). Many of them are addressed implicitly or not addressed at the early stages of a data stack. For example, data discovery isn’t a major issue when the data team consists of a single person. Still, it’s worth being aware of management considerations that arise as the data stack matures.
                                                                                                                                        • Timely automation. As a broad generalisation, engineers like automation. Erring on the side of automation is often a good idea, as it lets computers do what they do best and frees up human time to deal with things that have to be done manually. Done right, automation increases overall quality. However, creating automations takes time, e.g., if a monthly report takes five minutes to generate, and it’d take a day of coding to automate, it’s probably enough to write up the procedure to generate it. You have bigger fish to fry.
                                                                                                                                        • The need to be boring. Another trap that you can easily fall into is trying shiny new tools. Despite what vendors might say, it’s rare for new tools to be truly transformative. You should strive to be boring in your choice of components. Use proven tools and services for the minimum viable data stack, keeping the shiny experimental stuff to your hobby side projects (I learnt this the hard way).
                                                                                                                                        • Speed of iteration. Some people believe that high quality always comes at the cost of iteration speed. I disagree, for the same reasons Martin Fowler pointed out in an essay on how increasing the internal quality of software increases iteration speed within weeks of the start of a project. In short, if you don’t invest in quality, you’re committing yourself to spending much of your time firefighting as the complexity of the data stack increases. However, overthinking reversible decisions or spending too much time on non-critical issues is also a real possibility. Above all, you must remember that the goal of the data stack is to support business decisions, which may require some compromises to deliver value as rapidly as possible.

                                                                                                                                        Next: Choosing components

                                                                                                                                        As noted, I’m aiming for this to be the first in a series of posts on setting up a minimum viable data stack. Each future post will be dedicated to each of the key components, going deeper into currently-available tools for storage, ingestion, transformation, and analytics. The focus will be on tools that are sensible to use by startups.

                                                                                                                                        Stay tuned for future posts! In the meantime, feedback is always welcome.

                                                                                                                                        Subscribe
                                                                                                                                          -
                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/02/26/avoiding-ai-complexity-first-write-no-code/index.html b/2024/02/26/avoiding-ai-complexity-first-write-no-code/index.html index f8bf42675..f069a044d 100644 --- a/2024/02/26/avoiding-ai-complexity-first-write-no-code/index.html +++ b/2024/02/26/avoiding-ai-complexity-first-write-no-code/index.html @@ -2,7 +2,7 @@

                                                                                                                                          Avoiding AI complexity: First, write no code

                                                                                                                                          February 26, 2024
                                                                                                                                          Illustration showing ancient stone tablets with the inscription 'primum non codere' (inspired by primum non noncere: first, do no harm)

                                                                                                                                          Custom software is notoriously hard to build and maintain. Machine learning (ML) adds a layer of complexity on top of traditional software, with many novel ways to accrue technical debt. Therefore, my general advice to young startups considering custom ML development borrows from the first and second rules of optimisation:

                                                                                                                                          1. Don’t.
                                                                                                                                          2. Don’t Yet (for experts only).

                                                                                                                                          For startups where Data & AI/ML isn’t a core part of the team’s capabilities, there are usually higher priorities than building custom ML models. However, deriving commercial value from advancements in AI is still possible – even without writing code at all.

                                                                                                                                          I recently witnessed two stories that exemplify this point.

                                                                                                                                          Exhibit A. Consider this story: A technical lead reached out to me for advice on a computer vision project that wasn’t progressing as expected. For the sake of illustration, let’s say it was a custom model to classify food pictures as hotdog / not hotdog.

                                                                                                                                          The company had contracted an ML engineer to drive the project. Despite having no background in ML, the lead felt like the contractor was going down the wrong track, and asked me for my thoughts.

                                                                                                                                          It turned out that the contractor believed that the best path forward was trying different model architectures. My advice was to ensure the contractor did the data work first. Often, there are bigger gains to be had from data augmentations than from model tweaks (e.g., applying distortions to the hotdog photos). As the data wasn’t sensitive, I also suggested trying third-party computer vision APIs or GPT-4 Vision to get an idea of what’s possible with the dataset.

                                                                                                                                          Exhibit B. Recently, I caught up with an entrepreneur who comes from a marketing background. Just prior to our meeting, they had successfully pitched an app they had built to a large client.

                                                                                                                                          Remarkably, the app included a hotdog detector similar to the one the ML engineer was struggling to ship. The entrepreneur used the FlutterFlow no-code platform along with Google’s computer vision APIs to rapidly create an app with commercial value – without deep knowledge of ML.

                                                                                                                                          Expanding the rules

                                                                                                                                          No two companies are exactly alike. Sometimes, custom code or ML models are necessary. However, given the pace of innovation in no-code and low-code software & AI, starting with the least code possible is often a wise choice. Those who build software as their craft often have a blind spot when it comes to no-code possibilities – coders gonna code. However, it’s important to rein in the coding instinct. The difference in total cost between a custom build and using third-party APIs or no-code solutions can easily be in six or seven figures.

                                                                                                                                          When contemplating custom ML and AI development, consider the following options:

                                                                                                                                          1. Don’t build it.
                                                                                                                                          2. Wait until it becomes easier.
                                                                                                                                          3. Use a no-code solution.
                                                                                                                                          4. Get a software engineer to implement it with third-party APIs.
                                                                                                                                          5. Get a software engineer to implement it with third-party models that you self-host (with minimal customisation).
                                                                                                                                          6. Get the experts to build it: ML engineers, data scientists, and data engineers.

                                                                                                                                          You should be pretty certain that the cost of Option 6 is worth the investment. One way to get there is by starting with one of Options 3-5, thereby proving (or disproving) that there’s commercial value in the most expensive option. And when the time comes for Option 6, always do the data work!

                                                                                                                                          Subscribe
                                                                                                                                            -
                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/03/04/two-types-of-startup-data-problems/index.html b/2024/03/04/two-types-of-startup-data-problems/index.html index 691ea19a3..4e5c508f6 100644 --- a/2024/03/04/two-types-of-startup-data-problems/index.html +++ b/2024/03/04/two-types-of-startup-data-problems/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2024/03/04/two-types-of-startup-data-problems/machine-learning-system.webp 1333w," src=https://yanirseroussi.com/2024/03/04/two-types-of-startup-data-problems/machine-learning-system_hu39175a98128e26abad0a91492df5a70e_35082_800x0_resize_q75_h2_box_2.webp alt="Diagram showing ML Code as a tiny part of a larger machine learning system." loading=lazy>

                                                                                                                                            Source: Hidden Technical Debt in Machine Learning Systems

                                                                                                                                            Side note: LLMs and black-box APIs. If LLMs are a core part of the product, I still consider it to be an ML-centric startup. I’m not sure if the term LLMOps will catch on, but it has a lot in common with MLOps. Likewise, using LLMs for retrieval-augmented generation is similar to building recommender systems, i.e., ML-centric. The same reasoning applies to black-box ML APIs: If they form a core part of the product, it’s an ML-centric startup because you need to think of data and metrics early on.

                                                                                                                                            Examples from my past

                                                                                                                                            My employment history includes work with both ML-centric and non-ML startups. These examples may help clarify the differences between the two startup types:

                                                                                                                                            • ML-centric startup: After my PhD, I was a founding data scientist with a startup called Giveable, where the product was a recommender system for gifts. Giveable disbanded, but I took the codebase to Next Commerce – a company that had a few products in the e-commerce space. There, I led the team that turned Giveable into Hynt – a recommender system as a service.
                                                                                                                                            • Non-ML startup: I was the first data hire at Car Next Door (now Uber Carshare). Despite my fancy Head of Data Science title (data science was still hyped up at the time), I did a lot of engineering work – including data & analytics engineering. I also built ML-ish models of customer lifetime value, but it was too early in the company’s life for anything too sophisticated on the ML front.
                                                                                                                                            • ML work at a non-ML scaleup: After Car Next Door, I spent 4.5 years at Automattic. The company’s headcount grew about 3-4 times in my time there (from about 500 employees). This growth included investment in data and ML: One major project I worked on was ML pipelines to improve marketing performance (e.g., automatically target customers that are most likely to upgrade as a result of a well-timed email). However, I was also involved in data-intensive projects that didn’t include ML.
                                                                                                                                            • ML-centric product with a non-ML startup: After Automattic, I joined Orkestra to help them build a new product that had ML at its core. However, the company’s main product wasn’t an ML product, and I left on good terms when they pivoted to focus on their main offering.

                                                                                                                                            With Giveable/Hynt and Orkestra, attempting ML product development without thinking of MLOps wasn’t going to work. With Car Next Door and Automattic, the company’s success never depended on MLOps, so an incremental approach to using data and ML was viable.

                                                                                                                                            Closing thoughts

                                                                                                                                            While both ML-centric and non-ML startups face data problems, the centrality of data varies between the two. Trying to run an ML-centric startup without a solid grasp of MLOps and data engineering practices is a recipe for failure, while non-ML startups can get away with less-than-ideal data practices for a long time.

                                                                                                                                            Personally, I’m always on the lookout for better ways of explaining these differences and coming up with accessible terminology to help founders who are navigating the space. ML-centric and non-ML will do for now, but other suggestions are welcome!

                                                                                                                                            1. This is a fine example of an advantage of writing publicly. The initial version of this post didn’t include the qualification of “unless they’re building a data-intensive product” – I realised it was missing the following day. Perhaps a better classification is data-centric versus data-supported, but I’ll leave that to a future post. ↩︎

                                                                                                                                            Subscribe
                                                                                                                                              -
                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/index.html b/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/index.html index 93d467f0c..effb479ad 100644 --- a/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/index.html +++ b/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/index.html @@ -2,7 +2,7 @@

                                                                                                                                              Questions to consider when using AI for PDF data extraction

                                                                                                                                              March 11, 2024
                                                                                                                                              Decorative image showing a book and flying documents with an AI-themed overlay

                                                                                                                                              The jagged frontier of AI means that you can’t always know which tasks are within the capabilities of current models. One such task is the extraction of structured data from PDFs. While this is fully within the capabilities of humans, there are unique challenges in getting off-the-shelf AIs like ChatGPT to do it well. Variants of this task have come up repeatedly in my recent discussions and work, so I put together this summary of my understanding of when it makes sense to try to automate PDF data extraction with AI.

                                                                                                                                              This post is structured as a series of questions I’d ask about a proposed project. Actual answers will depend on the specific project.

                                                                                                                                              What is your budget?

                                                                                                                                              I assume there is some business value in the extracted data. Therefore, it’s possible to estimate the dollar value of automating the manual processes that are used to extract the data. There’s a big difference between an extraction process that takes a junior employee a couple of weeks per year and one that keeps data entry specialists busy year-round. The budget determines the tools that can be used: If it’s low (e.g., in the thousands), it’s probably only worth spending a few days assessing feasibility with off-the-shelf tools like the OpenAI APIs. If it’s higher (e.g., hundreds of thousands), paying AI engineers to build a bespoke system becomes an option.

                                                                                                                                              How sensitive is the data?

                                                                                                                                              If the data you’re working with isn’t sensitive (e.g., financial statements by public companies), you’re in luck: You can use the best AIs available. A month ago, it was GPT-4. Today, you have a bunch of other proprietary options. Tomorrow, who knows?

                                                                                                                                              If the data is sensitive and can’t leave your organisation’s systems, your options are more limited. Depending on other aspects of the problem, it may mean you’re better off waiting for better open models. However, given the rate of progress in the open source AI ecosystem, assessing feasibility is about as simple as with proprietary solutions. It’s just that right now, you won’t be using the most capable models.

                                                                                                                                              How complex are the PDFs?

                                                                                                                                              There’s a wide variety of documents out there. If the PDFs you’re working with can be converted to text accurately, the AI models have a good chance of being able to extract the data you’re after. If the conversion to text is insufficiently accurate, the AIs stand little chance of outperforming data entry specialists – they just don’t see the PDFs as well as we do.

                                                                                                                                              It’s worth spending some time on this question. For example, if you’re using OpenAI’s APIs and much of the data you’re looking to extract is contained in tables within PDFs, you can first test whether you can reliably retrieve and display specific tables. Sticking with the example of financial statements, there’s a big difference between this 124-page sample from Grant Thornton and this seven-page sample from the Australian Fair Work Commission. Prompting GPT-4 (via ChatGPT Plus) to extract a table from the former PDF produced output that was only loosely-connected to the actual content. By contrast, GPT-4 perfectly reproduced tables from the latter PDF.

                                                                                                                                              In general, perfect conversion of PDF tables to text appears to be an unsolved problem. For example, in a benchmark from last year, the best tool tested only had about 50% accuracy when applied to tables in scientific papers. However, the field keeps moving. You may get an accuracy boost by treating PDFs as images and using vision models, as recommended by the Unstructured library. Anecdotally, I found Unstructured’s table parsing to be too inaccurate, but when I used GPT-4 Vision on screenshots of the same tables it yielded much better results. It also outperformed OpenAI’s default PDF parser. Your mileage will definitely vary.

                                                                                                                                              Things get even more complicated if some of the data you’re hoping to extract is in graphs and other figures contained in the PDFs. Verifying that the PDFs are not too complex for the AI models is definitely worth doing before jumping into more elaborate data extraction tasks.

                                                                                                                                              Can the tokenised PDFs fully fit in the model’s context window?

                                                                                                                                              The complexity of the PDFs determines how well the AI models can see them. Their length determines how much of them the models can see at once. There’s nuance depending on whether the PDFs are fully converted to text tokens or to text and images, but the number of tokens that can be fed in (i.e., the context window) is limited with the current generation of AI models. Context windows are rapidly expanding, with Google recently releasing a million-token model (approximately 700,000 words), but there’s still a cost per token that you need to consider when building solutions.

                                                                                                                                              If you’re working with PDFs that are larger than your context windows, you’re probably going to use retrieval-augmented generation, i.e., break the documents into chunks and feed specific chunks into the model based on the query. With some tools and APIs, this may be handled for you (e.g., with OpenAI’s Assistants). However, it introduces a source of inaccuracies that may not be acceptable for your use case.

                                                                                                                                              What is your teaching approach?

                                                                                                                                              Assuming you’re satisfied that the AIs can see enough of the PDFs well enough to provide useful answers, it’s time to test different ways of teaching them about the data extraction tasks. The question of which approaches you can test is closely tied to your budget. But even with large budgets, it’s best to start simple and only attempt more complicated approaches if the simpler ones fail. In order of implementation complexity, key teaching approaches are:

                                                                                                                                              • Zero-shot: Describe the task and the expected output, provide a new PDF, and see if you get the expected output.
                                                                                                                                              • Few-shot: In addition to describing the task and expected output, also provide examples of past PDFs and their extracted outputs. Given context window limitations, this is only feasible with relatively short PDFs and simple outputs.
                                                                                                                                              • Fine-tuning: This goes beyond the sort of prompting that has become accessible to the general population via ChatGPT. The general idea is that you can get better results by teaching the underlying model about your expected inputs and outputs. Even if you don’t have machine learning experts on your team, you may get good results by following resources such as the fine-tuning guide by OpenAI. However, success isn’t guaranteed, so it’s important to manage expectations and budgets accordingly. It may well be the case that you’re better off waiting a few months or years for new AI models, rather than investing in fine-tuning experimentation. New models are likely to make lower-effort zero/few-shot results better.
                                                                                                                                              • Custom models: Taking a step beyond fine-tuning, building custom machine learning models may be a good match for your budget and available expertise. However, you definitely won’t be doing it to automate a low-cost-low-frequency data entry process.

                                                                                                                                              Implicit in the above is the availability of some training & testing data (i.e., input PDFs and expected outputs). That is, no matter what approach you follow, you’d want to have some confidence that it works beyond a few test samples – use a large representative dataset to gain confidence in your solution.

                                                                                                                                              Can the AI model understand the input structures?

                                                                                                                                              This is closely related to the question of PDF complexity, but worth considering separately. I’m anthropomorphising a bit by talking about AI understanding, but just as they can’t see the same as we do, their level of understanding may also be unintuitive. For example, a recent paper that proposed a fine-tuning approach to improve GPT-3.5’s table understanding made the case that general language models can’t read tables reliably because:

                                                                                                                                              Natural language texts are (1) one-directional, (2) read left-to-right, where (3) swapping two tokens will generally change the meaning of a sentence. In contrast, relational tables are (1) two-dimensional in nature with both rows and columns, (2) where reading top-to-bottom in the vertical direction for values in the same column, is crucial in many table-tasks. Furthermore, unlike text, (3) tables are largely “invariant” to row and column permutations, where swapping two rows or columns do not generally change the semantic meaning of the table.

                                                                                                                                              This argument is compelling, but given the emergent abilities of large language models and the fact that we no longer know what goes into proprietary models beyond GPT-3.5, I wouldn’t bet on these limitations being an issue for all tabular data. Again, experimenting with your specific use case is key. If you encounter issues, it’s worth probing the models to check if they exhibit any semantic understanding beyond just reproducing the inputs.

                                                                                                                                              Can the AI model understand and produce the output structures?

                                                                                                                                              If you’re building custom models, it’s straightforward to get exactly the output structure you want (e.g., a complex JSON). Otherwise, if you’re prompting a language model, you need to ask nicely and hope for the best. That said, there are strategies to get models to produce the output structures you want, such as using OpenAI’s function calling or the Outlines library. However, as with the example of tabular inputs, there is a difference between being able to produce an output that conforms to a specific output schema and populating the schema with values that make semantic sense. Breaking down complex outputs to simpler structures and using prompt chaining may be helpful in some cases.

                                                                                                                                              What is your long-term validation approach?

                                                                                                                                              Assuming you successfully build an AI solution that can replace manual data entry, should you completely stop manual data extraction? As with other questions, it depends on the use case, but it’s worth considering a gradual switch to full automation. For example, you can keep the manual process for 10% of new data to verify that the whole system works as expected. This is especially worth doing when working with publicly-available datasets, as there’s a non-zero chance that the models you’re using have seen the input training data before (though they probably haven’t seen your outputs).

                                                                                                                                              Anything else?

                                                                                                                                              If I missed important questions, please let me know and I will update this post.

                                                                                                                                              Subscribe
                                                                                                                                                -
                                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/index.html b/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/index.html index bc6d902ce..078b3b153 100644 --- a/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/index.html +++ b/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/lord-howe-island-algal-hole-trevally_hub04ed3d37235d405b5867447ff565938_331694_1500x0_resize_q75_h2_box_2.webp 1500w," src=https://yanirseroussi.com/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/lord-howe-island-algal-hole-trevally_hub04ed3d37235d405b5867447ff565938_331694_800x0_resize_q75_h2_box_2.webp alt="Automation or no automation, hanging out with the fish is always fun" loading=lazy>

                                                                                                                                                Automation or no automation, hanging out with the fish is always fun

                                                                                                                                                Subscribe
                                                                                                                                                  -
                                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/index.html b/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/index.html index 3e7272dfe..766a3b645 100644 --- a/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/index.html +++ b/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/index.html @@ -7,7 +7,7 @@ https://yanirseroussi.com/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/work-on-climate-six-month-milestone-award.webp 1200w," src=https://yanirseroussi.com/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/work-on-climate-six-month-milestone-award_hu8ed3990140315acddae2a6c763f03c66_143390_800x0_resize_q75_h2_box_2.webp alt="Six-month milestone award from Work on Climate, recognising my contribution to the organisation." loading=lazy>

                                                                                                                                                  This milestone award is another example of Work on Climate’s similarity to full-time work environments. It’s nice to be recognised!

                                                                                                                                                  Subscribe
                                                                                                                                                    -
                                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/index.html b/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/index.html index e985e7209..5c6914c61 100644 --- a/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/index.html +++ b/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/index.html @@ -2,7 +2,7 @@

                                                                                                                                                    AI does not obviate the need for testing and observability

                                                                                                                                                    April 15, 2024
                                                                                                                                                    clunky untested bot on the left, slicker bot on the right

                                                                                                                                                    The excitement sparked by ChatGPT has led to a flood of funding for building AI applications, especially around large language models (LLMs). The ease of getting started with AI can lead to excessive enthusiasm, to the point of believing that we have entered a new regime of software development where old best practices no longer apply. The goal of this post is to demonstrate that we are still in the old regime: Testing and observability remain key to AI success beyond initial prototypes.

                                                                                                                                                    Bookmark and reuse if anyone tries to claim otherwise.

                                                                                                                                                    First, let’s acknowledge the fact that prototyping AI applications is now easier than ever. For example, I recently watched this video by Hrishi Olickel, which demonstrates how to go from zero to a working AI-powered app in about thirty minutes. Examples like this abound, but I have a feeling that people might miss two key messages from the video:

                                                                                                                                                    1. 99% of the time, the problem is with your data.
                                                                                                                                                    2. The app isn’t ready for production.

                                                                                                                                                    Two elements that solid production-level apps include are testing and observability. This is highlighted in recent posts by two consultants who are helping companies ship LLM-powered applications:

                                                                                                                                                    1. Your AI Product Needs Evals by Hamel Husain. Key quote: “Unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems.”
                                                                                                                                                    2. Levels of Complexity: RAG Applications by Jason Liu. Level 3 is observability. Level 4 is evaluations.

                                                                                                                                                    The use of the word evaluations (or evals) by both authors is intentional. This is the common term for testing that deals with the challenges of working with LLMs (essentially a complex mapping from any text input to any text output). As noted in the OpenAI Evals repository:

                                                                                                                                                    If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case.

                                                                                                                                                    That is, we are at the opposite to a new regime where traditional software testing can be forgotten: Production-level AI apps still require all the usual software tests, as well as AI-specific evaluations.

                                                                                                                                                    In a way, this is nothing new. Before ChatGPT drew significant attention to LLMs, much of the buzz was around traditional machine learning (ML) apps. And many of the best practices from ML engineering apply to LLM / AI engineering.

                                                                                                                                                    If you are inexperienced with shipping production-grade AI/ML/LLM applications, please don’t let it stop you from prototyping. But if you are getting serious about going beyond a prototype, it’s time to either get help from experienced AI engineers, or to become one yourself (experience is a great teacher). Just remember that there is no way around testing and observability if you want to ship a quality product.

                                                                                                                                                    Subscribe
                                                                                                                                                      -
                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/04/22/assessing-a-startups-data-to-ai-health/index.html b/2024/04/22/assessing-a-startups-data-to-ai-health/index.html index 23539539a..5e2d89663 100644 --- a/2024/04/22/assessing-a-startups-data-to-ai-health/index.html +++ b/2024/04/22/assessing-a-startups-data-to-ai-health/index.html @@ -2,7 +2,7 @@

                                                                                                                                                      Assessing a startup's data-to-AI health

                                                                                                                                                      April 22, 2024
                                                                                                                                                      a person dressed as a doctor conducting a health check on a screen

                                                                                                                                                      In the past year, I went from exploring product ideas to committing to my current consulting practice. One thing that became apparent was that I needed to get better at communicating my unique value proposition: who I serve, and how I can help them. The circus that is data (/AI/ML/BI/analytics/…) titles and terminology definitely doesn’t help. Sprinkle a couple of decades of hype cycles on top, and you end up where we are today: a mess of inflated expectations followed by disappointments. But also a wealth of opportunities to generate genuine value.

                                                                                                                                                      Anyway, I’m now fairly clear on whom I’m actively targeting: funded Australian startups (around seed to series A) in the climate & nature tech space, who can use help on their data-to-AI journey. I started calling it data-to-AI rather than data & AI or data/AI/ML because anything AI (/ML/data science/…) or analytics starts with data – and keeps going back to data.

                                                                                                                                                      How I can help isn’t as clearly communicated as I’d like it to be, so I’ve been working on that. Offerings at the cheap and expensive ends of the spectrum are easy to explain: one-off advisory calls include bespoke on-the-spot advice, while fractional chief data & AI officer engagements include similar responsibilities to those of a full-timer with the same title. However, it’s in nobody’s best interest to jump straight into a fractional relationship. To address this, I’ve been working on a standard offering that’d be more structured than advisory calls, deliver value to the client, and allow both parties to uncover opportunities and see how we work together.

                                                                                                                                                      My working title for the offering is Data-to-AI Health Check (better suggestions welcome). The idea is to assess where the startup stands with their data/AI/ML stack and capabilities, and identify the top opportunities for improvement.

                                                                                                                                                      This has been on my mind for a while, so I’ve collected a heap of documents and questions for inspiration. I’m now at the “too overwhelmed” phase of turning it into something I can present, but hopefully I’ll have it all sorted in the coming weeks.

                                                                                                                                                      In the meantime (and in the spirit of building in public), the rest of this post describes the areas I think are most important to assess. Suggestions for areas I might have missed are welcome. In future posts, I’ll add more detail on performing the assessment, which will undoubtedly evolve as I offer it to more clients.

                                                                                                                                                      Assessment areas

                                                                                                                                                      Product and business model. Understanding what the startup is about and where it’s going is key to understanding where data/AI/ML fit in. One useful lens is determining whether the product is ML-centric or non-ML, with non-ML products varying in their data intensity from data-centric to data-supported. It’s also important to understand key metrics and how they’re measured.

                                                                                                                                                      People. Who’s working for the company and what is the team structure? In particular, what are the current data/AI/ML capabilities and experience? Can the current staff deliver what the business needs? If there are skill gaps (e.g., they haven’t yet made their first data hire), what’s the plan to address them? Can the current team adequately assess the skills of data people?

                                                                                                                                                      Processes and project management. The best people will fail to deliver projects if the company’s processes have deep flaws. My general opinion is that all the best practices from software development can and should be applied to data projects (e.g., see posts from 2023 and 2018). However, data entropy and the probabilistic nature of AI/ML require extra care and practices in addition to traditional software development.

                                                                                                                                                      Culture. Knowing what people are on the team and what processes are in place isn’t enough to assess how well the team can deliver the product vision. Culture – the unwritten norms and beliefs of the company – matters. A lot. For example, if the founder doesn’t tolerate data-backed evidence that contradicts their preconceived notions, it’s likely to be an impediment to data/AI/ML project delivery. Similarly, it’s worth paying attention to how experiments are treated: If a hypothesis behind an experiment turns out to be unsupported, it’s not a failure. Failing to learn from experiments is the true failure.

                                                                                                                                                      Data. What data is the company dealing with? What are the data’s volume, velocity, and variety? Is all the necessary data being captured? How clean is it? Where is it stored and how is it processed? What data management practices are in place, both explicitly and implicitly?

                                                                                                                                                      Tech. Closely related to data is the tech architecture, systems, and software. Tech includes where the data lives and how it flows, particularly how it feeds into AI/ML/analytics applications. Of particular interest is the allocation of innovation tokens. Innovation tokens should be spent on tech that makes the startup meaningfully unique to its customers. Everything else should be boring and standard, i.e., proven to work and fit for purpose.

                                                                                                                                                      Security and compliance. Security is interwoven through all of the above. For example, you want a culture where any person can flag security risks – some of which may only be visible if you’re close to the code and data. Security breaches and data leaks can destroy companies, especially young startups that haven’t earned customer trust yet. Particular attention should be paid to compliance issues that arise with data collection, e.g., around personal and regulated data.

                                                                                                                                                      Other opportunities and risks. In exploring the above areas, issues that don’t fit neatly into any bucket are likely to be uncovered. These may be new opportunities or risks. It’s important to keep an eye out for such cases and flag them accordingly.

                                                                                                                                                      Closing thoughts

                                                                                                                                                      In my experience, it’s easy to find a thousand areas for improvement once you become familiar with a startup or a large company division. It’s harder to identify the top three items to work on next – it is a bet on the highest-impact items that are feasible to deliver.

                                                                                                                                                      It is also a challenge to distill the Data-to-AI Health Check to a set of questions that would probe the right areas without burdening the startup too much. I’ll report back once I’ve figured it out. In the meantime, comments are welcome!

                                                                                                                                                      Subscribe
                                                                                                                                                        -
                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/04/29/mentorship-and-the-art-of-actionable-advice/index.html b/2024/04/29/mentorship-and-the-art-of-actionable-advice/index.html index 6cd5d758f..a94afe1f1 100644 --- a/2024/04/29/mentorship-and-the-art-of-actionable-advice/index.html +++ b/2024/04/29/mentorship-and-the-art-of-actionable-advice/index.html @@ -2,7 +2,7 @@

                                                                                                                                                        Mentorship and the art of actionable advice

                                                                                                                                                        April 29, 2024
                                                                                                                                                        ChatGPT's depiction of a robot mentoring a robot

                                                                                                                                                        One of my challenges with the transition to solo consulting is learning to deliver timely, actionable advice. It’s usually easy for me to identify many areas for improvement. Distilling a long list of “obvious” opportunities to the top items that would make a difference is harder. And the hardest thing is packaging it all up as timely advice that people can act on.

                                                                                                                                                        To help address this challenge, I recently joined EnergyLab and GrowthMentor as a mentor. EnergyLab is Australia’s largest climate tech startup accelerator, while GrowthMentor is an international platform for mentorship around startup growth. Both are relevant to my focus on helping leaders of climate/nature tech startups ship data-intensive solutions (including AI/ML, data science, and advanced analytics – this stuff goes by many confusing and hyped-up names).

                                                                                                                                                        The rest of this post presents some of my reflections on packaging advice and expertise. I’m always happy to discuss these topics and connect directly with people I may be able to help, so please feel free to reach out with feedback.

                                                                                                                                                        Actionable and timely advice by example

                                                                                                                                                        We all know we should get enough sleep. But telling a busy insomniac with young children to “sleep more” isn’t actionable. It’s more helpful to provide them with specific strategies for improving their sleep hygiene, like keeping screens out of their bed. And the more specific, the better: “At 10pm, put your phone to charge in another room” gives them exactly one thing they can do tonight.

                                                                                                                                                        There is more to it, though. If your insomniac friend comes to you complaining about having a bad night, they’re probably not expecting advice on where to charge their phone – at least not in that specific moment. The timing of the advice can make all the difference between them following it and them doing nothing (or worse – getting annoyed by your lack of empathy).

                                                                                                                                                        The same goes for advising anyone about anything.

                                                                                                                                                        In my case, advice of a general nature like “your data should be clean, relevant, and plentiful” is nice – but it’s also kinda useless. Getting more specific on strategies and tools is better, e.g, “consider dbt to manage and test data transformations”. Getting to the root of what they want to achieve may yield completely different advice, like “don’t worry about dbt for now, if you want that ML project you mentioned to succeed, you need to instrument and start collecting data around feature X of your product as soon as possible.”

                                                                                                                                                        On listening and packaging expertise

                                                                                                                                                        To get to a point of giving timely, actionable advice, you need more than functional expertise. It’s important to listen to what the other person is saying (and not saying), and figuring out what they’re most likely to respond to. This is easier with people with whom you’ve already built a relationship than with new acquaintances – which makes the challenge of mentoring at scale all the more interesting.

                                                                                                                                                        One key aspect is aligning on expectations. Coming at it from the mentor side, I aim to be transparent about where I can and cannot help, so as to only attract mentees who are likely to be a good fit. However, after almost twenty years in the tech industry and over a decade in data / AI / engineering roles with startups & scaleups, it’s hard to succinctly describe my area of expertise. For example, I liked the label data scientist when it became popular around 2012, but both the label and I have changed over the years. There are major differences between my experience and that of a new data scientist who is fresh out of university. Me using a commodity label like data scientist is not in anyone’s best interest.

                                                                                                                                                        Aligning on expectations is easier in close long-term relationships. In our professional lives, such relationships are commonly formed when working for one employer at a time. Indeed, most of my work experience was that of an employee. And like many employees with long-term roles, it was easy for me to identify opportunities for improvement and provide actionable advice to my colleagues. There is a lot of implicit listening going on when you are dedicated to a single employer!

                                                                                                                                                        In the absence of a long-term relationship, it’s important to communicate expectations ahead of time. For example, this is what I put in as my “support offered” for EnergyLab founders:

                                                                                                                                                        Advice on data strategy, data hiring, AI/ML projects, data science, advanced analytics, and data-intensive solutions.

                                                                                                                                                        I have over a decade of experience in data / AI / engineering roles with Australian startups (most famous: Car Next Door / Uber Carshare & Orkestra), international scaleups (Automattic / WordPress.com), and big tech (Intel / Qualcomm / Google). This means I also have many opinions on tech and startups beyond my specific expertise, which may be of use to some founders. :)

                                                                                                                                                        In a case of a potential fit, the next step on my end is to listen. My aim is to only offer mentorship in situations where I add value. Redirecting founders to others in my network who may be a better fit than me is a better outcome than attempting to give advice on topics that fall outside my area of expertise.

                                                                                                                                                        True experts are always learning

                                                                                                                                                        Another key aspect of providing advice as a mentor/expert is recognising that no one knows everything. Even within narrow areas of Data & AI, things are moving so fast that even the most knowledgeable people have no chance of keeping up.

                                                                                                                                                        However, expertise is a relative term. I know more about shipping data-intensive solutions than a non-technical CEO, so I can probably help them (especially if they don’t have in-house data experts). I know less about PyTorch internals than an ML engineer who has been focused solely on deep learning for the past decade, so I’ll defer to such experts when deep PyTorch expertise is needed.

                                                                                                                                                        As another analogy, consider a general practice doctor named Amy – she is a medical expert in comparison to most of the population. But Amy wouldn’t try to perform brain surgery – she’ll refer you to a neurosurgeon (Barbara?), who is an expert in comparison to Amy.

                                                                                                                                                        Things are fuzzier in the unregulated software and data worlds. Memorably, the young child of a past manager one day announced: “My computer has data on it! I am a data scientist!” The equivalent of such pronouncements in the adult world was the swift shift of LinkedIn titles in the years after 2012 – peak data science hype. By contrast, declaring yourself a medical doctor will land you in prison in many countries.

                                                                                                                                                        In the absence of regulated data expertise (which is probably undesirable), we are left with heuristics for determining who should be providing data advice. One of my favourite heuristics aligns with GrowthMentor’s core value of humility. In their words: “Nobody knows everything and we should all be open to hearing a different perspective on what we are working on. […] Opening yourself up to feedback from your peers will not only make you a stronger person, but also lead to more confidence in your professional life.”

                                                                                                                                                        To me, this is the sign of a true expert: Knowing that you still have a lot to learn. And this brings me back to what I’m aiming to learn and improve through mentorship: Giving timely, actionable advice outside the context of employee-employer relationships.

                                                                                                                                                        I’ll report back on how it goes in the future.

                                                                                                                                                        Subscribe
                                                                                                                                                          -
                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/index.html b/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/index.html index b046378c1..1f328bbd6 100644 --- a/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/index.html +++ b/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/index.html @@ -2,7 +2,7 @@

                                                                                                                                                          Business questions to ask before taking a startup data role

                                                                                                                                                          May 6, 2024
                                                                                                                                                          black box with a question mark

                                                                                                                                                          If you join a startup as an early employee, you’re essentially an investor. But unlike capital investors, you can’t diversify your portfolio as a full-timer. You need assurances that your time investment is likely to yield a positive return. Ideally, this would be a better return than the return on any other use of your work time.

                                                                                                                                                          Good startups guarantee a return for early employees by paying a competitive base salary. Typically, compensation also includes equity in the company, which vests over time. However, unlike equity in publicly-traded companies, there’s a good chance that your startup equity will be worthless.

                                                                                                                                                          To help you assess the value of your startup equity, this post presents questions from the Product & Business Model section of my Data-to-AI Health Check for Startups. In creating the health check, I realised I could have titled it “questions I should have asked past employers”, i.e., you get to learn from my mistakes! The health check has seven other areas, which I may cover in future posts.

                                                                                                                                                          Before proceeding, note these assumptions:

                                                                                                                                                          1. You don’t need to take a specific job offer urgently, i.e., you have multiple options or time to search for better options. If there’s urgency, you can take a suboptimal role, build your skills and savings, and aim to generate better options on your next search.
                                                                                                                                                          2. You’re not seen as too junior to be asking deep questions. If you’re early in your career, consider if early-stage startups are for you: My view is that you may be better off working with an established company, where there are more structured mentorship opportunities.
                                                                                                                                                          3. The startup isn’t proposing an unpaid equity-only position. If you’re expected to work full-time without a salary, you’re a founder.
                                                                                                                                                          4. The startup is small enough for you to speak with the founders as part of the recruitment process (maybe <100 employees).
                                                                                                                                                          5. The founders aren’t refusing to answer your questions. If they are, move on.

                                                                                                                                                          Investor-level product & business questions

                                                                                                                                                          Most of the following questions are typically answered by a pitch deck, so asking the founders to take you through the pitch may be the most time-efficient way of getting answers. Depending on the stage of product development, you may also be able to gather some answers yourself from the company’s website.

                                                                                                                                                          While these are investor-level questions, your assessment of the answers should be different from that of an investor. You don’t have dozens of other startups in your portfolio – only one. Better make sure it’s a good one.

                                                                                                                                                          Q1: What is your company’s purpose? What problem are you solving and why? You’re looking for a plausible story and solution. It’s important to understand founder motivations, and assess whether you want to spend many of your waking hours bringing their vision to life.

                                                                                                                                                          Q2: What does your product do? If the product is already live, a demo would be best to answer this. Otherwise, wireframes and other plans would be good enough. If the product isn’t live yet, watch out for unrealistic plans. You don’t want to work on a product that will never get released.

                                                                                                                                                          Q3: What are the relevant market sizes (TAM/SAM/SOM)? TAM is total addressable market – the total market demand for the product, which helps assess the growth potential. SAM is serviceable addressable market – the market demand the product can plausibly fulfill, which helps assess revenue targets. SOM is serviceable obtainable market – the part of the market that the startup’s product can capture, which helps assess short-term growth potential.

                                                                                                                                                          Q4: Where do the problem, market, and solution sit on Jason Cohen’s problem flowchart? While TAM/SAM/SOM are useful high-level metrics, the problem flowchart goes deeper into assessing the viability of a startup. A surprising number of startups skip such assessments, and fail as a result. You may regret joining startups that make such preventable blunders.

                                                                                                                                                          Q5: What is on the product roadmap for the next 6-12-24 months? In startup-land, plans for 12-24 months are in the realm of wishful thinking, but it’s good to have an idea of the general direction. Knowing what’s on the roadmap for the next six months will help you assess whether you want to come on board.

                                                                                                                                                          Q6: What is the business model, i.e., how do you make money? Together with the other questions, this will help you assess the viability of the business. You should be especially wary if the founders haven’t figured out how to generate revenue yet, which means they’ll have to raise money to keep paying you. If they’re not seeing healthy growth in other key metrics (e.g., number of active users), they’ll struggle to raise more funding.

                                                                                                                                                          Q7: What is the competition? How’s your product differentiated in the eyes of customers? How hard is it for competitors to copy you? Founders should have solid knowledge of the competitive landscape, and be able to explain why customers choose their product over the competition – and why they’ll continue to do so. Steer clear of founders who exhibit a low understanding of customer wants and needs. The company’s value ultimately comes from making something people want.

                                                                                                                                                          Q8: What are the key business metrics (definitions, values, and trajectories)? This is especially pertinent if you’re the type of data person who’s going to get deep into business metrics as part of your job (a data scientist/analyst, as opposed to a data/AI/ML engineer). But regardless of role, it’s important for you to know how the business is performing. You should be confident that startup executives are measuring the right things.

                                                                                                                                                          Q9: Since the last raise, how has the company performed against its goals? This includes goals that are covered by the key business metrics, as well as product development milestones. Repeatedly failing to achieve self-imposed goals is often a red flag – the goals may be unrealistic, and the business may not be viable.

                                                                                                                                                          Q10: How much runway is left before another raise is needed? This is critical for employees to know. For example, if there are only three months left before the startup runs out of money, you may be out of a job pretty quickly. Note that the question still applies if the startup is bootstrapped (i.e., self-funded or funded by revenue) – money needs to come from somewhere to cover your salary.

                                                                                                                                                          Data-to-AI product & business questions

                                                                                                                                                          While the above questions should be asked by any early startup employee, you should also get answers for the following questions if you’re considering a data/AI/ML role. If you’re the first data hire, pay specific attention to answers that indicate that the startup isn’t ready for a data hire, or that you may have to wear hats you’re unwilling to wear. For example, if you’re passionate about advanced AI/ML modelling but there are gaps in data engineering and basic analytics, you’re likely to be the one doing the data work to address those gaps.

                                                                                                                                                          Q11: What is the data intensity of the product on a scale of 1-5? High data intensity typically requires low-latency processing of large volumes of data with more than one database server. With high intensity, a slowdown in data processing would noticeably affect key business metrics. High data intensity means that solid data engineering skills are required for success – it’s important to ascertain that founders are aware of this requirement.

                                                                                                                                                          Q12: Is advanced AI/ML core to the product? What if you used simple heuristics? One issue with AI/ML is the hype. AI is indeed transformative and exciting, but using AI isn’t always required for the product to succeed. In the words of Google’s first rule of ML: “Don’t be afraid to launch a product without machine learning”. As a data professional and an outsider, you are in a good position to assess whether advanced AI/ML has to be core to the product. The answer should only be yes if it would make a difference in the eyes of the customers. Using AI/ML too early is often a premature optimisation. You should assess whether the added complexity of dealing with MLOps is justified.

                                                                                                                                                          Q13: Are you planning to increase data intensity or advanced AI/ML use? Why? This question is similar to the one about the product roadmap, but specific to data/AI/ML. Again, the Why is key – ensure that there’s a solid business case for increased data/AI/ML complexity. In a healthy startup, increased complexity is driven by customer need, not by excitement about shiny tech.

                                                                                                                                                          Q14: Are any decisions routinely blocked or delayed by limited access to data? This question helps assess gaps in data collection and quality, as well as the company’s culture around the use of data. It should also help you understand what sort of work is likely to be needed, e.g., even if there are plans to use more advanced AI/ML, the reality of data gaps may mean that plenty of data engineering work is needed.

                                                                                                                                                          Feedback welcome

                                                                                                                                                          If you found the above questions helpful or if you have any other feedback, I’d love to hear from you. I’m planning to evolve my Data-to-AI Health Check over time and post more on the other areas you should ask about. Subscribing for updates is the best way to get notified when it happens.

                                                                                                                                                          Subscribe
                                                                                                                                                            -
                                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/index.html b/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/index.html index 3f310604c..c84bd0354 100644 --- a/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/index.html +++ b/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/index.html @@ -2,7 +2,7 @@

                                                                                                                                                            Probing the People aspects of an early-stage startup

                                                                                                                                                            May 13, 2024
                                                                                                                                                            startupy people in a startupy space around a massive question mark (good old ChatGPT...)

                                                                                                                                                            A successful startup is fundamentally a group of people who execute well together. Building a viable product is key, but changing the product is easier than changing the founders. If you’re considering a role at an early-stage startup, you will be investing your best waking hours in the company. It’s important to assess the team before making such an investment.

                                                                                                                                                            To learn about the team, ask questions from the People section of my Data-to-AI Health Check for Startups. Some questions emphasise probing Data/AI/ML capabilities – an area that’s often misunderstood by non-specialists. However, this emphasis can be shifted to different functional areas as needed. Similarly to my previous post on scrutinising the Product & Business Model, the rest of this post lists my questions along with brief opinionated explanations.

                                                                                                                                                            People questions

                                                                                                                                                            Q1: Who are the founders? What are their skills and experience? Founders make or break a startup. It’s important to gain confidence that they have the skills and experience required to build the company, along with the mindset needed to keep learning and developing relevant skills. Founders who were previously successful are an especially positive sign – it indicates that they have the persistence and flexibility needed to build a business.

                                                                                                                                                            Q2: What motivates the founders? How passionate are they about the startup’s problem space? My favourite founders are those who build a business based on their deep understanding of customer problems in an area they deeply care about. For example, I previously worked with Orkestra – a software-as-a-service startup that grew directly out of the founders’ experience as consultants. In Orkestra’s case, the founders had already spent years working together solving customer problems prior to founding the company. By contrast, some startups are founded by near-strangers just because the founders want to build something – a red flag.

                                                                                                                                                            Q3: Have any founders left? Why and how? Startups can turn friends into foes. Foes holding a significant share of the company may lead to its destruction. But even in cases where founders leave on good terms without significant equity, their departure stories help understand founder personalities and the trajectory of the company. For example, if you’re considering a full-time position, and the story of how the remaining founders treated departing founders gives you pause, you’re better off working elsewhere.

                                                                                                                                                            Q4: Who are the key employees? Early employees are almost as important to startup success as founders. In fact, I know of multiple cases where employees “became” founders even though they weren’t there from day one. For example, after my PhD I joined Giveable as a founding data scientist. As the first employee, I was in charge of building the backend for Giveable’s B2C gift recommendation web app. Due to market conditions, we pivoted to a B2B recommender-as-a-service offering – not what the original founder had envisioned. He decided to move on, and I was left with much more equity than originally planned, along with the rights to the code. While I could have kept going as a “founder”, I decided to use the codebase to continue building the same B2B product as part of a more established ecommerce startup.

                                                                                                                                                            Q5: Have any key employees left (including involuntarily)? Why and how? With early employees being almost as important to startup success as founders, stories of their departures can be as informative as stories of founder departures. If you’re considering a startup job, these stories can tell you a lot about founder-employee dynamics, before you become an employee. If you’re especially thorough, you can even reach out to the former employees to get their side of the story. Positive signs include low employee turnover and founders who are comfortable with you speaking to their former employees.

                                                                                                                                                            Q6: How committed are the founders and key employees (partly measured by work time spent on the startup)? Early on, it’s common for founders and employees to be involved on a part-time basis. This is fine, but if you’re going to commit a significant chunk of your time to the startup, you need to know who you’ll be working with. As an employee, you can usually ignore the big names that are listed as advisors on the startup’s website – their involvement is typically minimal. That said, both advisors and fractional contractors provide access to expertise and connections that may not be necessary on a full-time basis. In fact, fractional help is much better than premature hiring, which unnecessarily burns through funding. The main things to look at in an answer to the commitment question are: (1) transparency; and (2) that committed staff have the skills needed to achieve the next milestones.

                                                                                                                                                            Q7: What hiring practices do you follow? How do you assess the skills of new experts (e.g., first data hire)? Given the importance of early employees, a loose hiring process is a cause for alarm. However, thoughtlessly borrowing hiring practices from the likes of Google is also problematic, as such processes are laughably hackable and tedious to everyone involved. Startups can and should move faster on hiring than established players: My favourite hiring processes include paid work on real problems after an initial low-cost filter. These are hard to scale, but there’s no need to scale hiring in the early days. Paying for work on real problems also helps address the challenge of assessing the skills of new experts – they are judged on real work output rather than on confidence, pedigree, and performance on convoluted tasks.

                                                                                                                                                            Q8: Do you pay market rates? Startups that don’t pay market rates are best avoided. They’re unlikely to attract and retain quality employees. Founders of such startups may also fall victim to classic fallacies like the 1975 Mythical Man-Month, and make expensive mistakes like hiring two mediocre engineers in place of one excellent engineer. When it comes to software (and data) development, higher quality often incurs a lower overall cost. Paying market rates and hiring great people is the way to go, especially in the age of AI-powered interns.

                                                                                                                                                            Q9: Are there any critical skill gaps among current personnel (especially around data/AI/ML)? If you’re asking this question as a candidate, you’re probably going to fill one of the gaps. However, gaps are relative to what the startup is trying to do. For example, if they have ambitious AI/ML plans that require a range of data skills they don’t have on the current team (from data engineering through data science to AI/ML engineering), they better be planning to hire more than one junior data generalist.

                                                                                                                                                            Q10: What’s the hiring roadmap for the next 6-12-24 months? How will it affect the runway? Is it dependent on new funding or revenue growth? Startup founders usually have grand plans – that’s what you want from founders! But plans for 12-24 months are often in the realm of wishful thinking, and a lot can change even in six months. As a candidate, try to get a realistic view of the hiring that is highly likely to happen, along with the hiring that is dependent on new money coming in. Assuming that the latter doesn’t happen due to a cashflow crunch, would you still take the job?

                                                                                                                                                            Even more questions?

                                                                                                                                                            This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                            You can download a guide containing all the questions as a PDF. The next area I’ll cover is Culture – how people work together. Feedback is always welcome!

                                                                                                                                                            Subscribe
                                                                                                                                                              -
                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/index.html b/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/index.html index ee0cb0b87..37590fc47 100644 --- a/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/index.html +++ b/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/index.html @@ -2,7 +2,7 @@

                                                                                                                                                              Question startup culture before accepting a data-to-AI role

                                                                                                                                                              May 20, 2024
                                                                                                                                                              an illustration of the data 'garbage in, garbage out' concept

                                                                                                                                                              AI shmAI. If you’ve been paying any attention, you’d know that for the vast majority of AI/ML projects, the real value comes from the data.

                                                                                                                                                              And if you’re considering a role with a startup that has grand AI plans, you’d better learn about the startup’s work culture – which includes its data culture.

                                                                                                                                                              To help you, this post discusses the questions from the Culture section of my Data-to-AI Health Check for Startups. Let’s jump right into it.

                                                                                                                                                              Q1: How often are people expected to work outside normal business hours (founders included)? Is unreasonable overtime compensated? “It’s a startup” isn’t a valid excuse for constant overwork. Most successful startups require a sustained effort over many years, and working at capacity reduces productivity over time. That said, high-effort spikes are inevitable – but unusual efforts should be recognised and compensated.

                                                                                                                                                              Q2: Do people go on leave regularly (founders included)? This probes a similar cultural norm around overwork to Q1. Stay away from places where people never go on leave. It leads to burnout and collective stupidity: Knowledge workers need downtime to take a step back and come up with new creative ideas. Humans are not AIs.

                                                                                                                                                              Q3: How do employees view the leadership team and founders? A small startup won’t have significant quantitative data on employee views. Even at larger companies, employee surveys are often designed and administered in a way that masks problems. If you’re considering a role with a startup, ask to speak with current employees to learn about their views on the culture, founders, and the company’s prospects. If the company is established, sites like Glassdoor and Blind can help you probe issues beyond the current employee base. In any case, remember that there’s an inherent selection bias in only questioning current staff, which is why it’s worth learning about former employees and ex-founders.

                                                                                                                                                              Q4: How are wins celebrated? How are failures and mistakes analysed? The unfortunate reality is that many startup founders have little experience running or working at a startup. Therefore, they may not appreciate the need to celebrate wins or to take time to learn from failures. Rather than asking about general rituals, you could ask for examples: What did you do after the last big release? What did you learn from the latest outage? How will you mitigate similar outages?

                                                                                                                                                              Q5: How is excellent/poor individual performance evaluated and handled? If you’ve worked anywhere at any capacity, you’d know that the following is true: (1) underappreciated excellent employees may leave; and (2) poor performers may drag an entire team down. Before you join a growing startup, it pays to know that founders have put some thought into performance management – especially if excellence is one of your core values.

                                                                                                                                                              Q6: Does the company run data-informed experiments (like A/B tests)? If so, what is considered a successful experiment? For example, what happens if a well-run experiment produces results that contradict the CEO’s opinion? Finally, a question that directly addresses data culture! If you are considering a data role, a culture of intelligent experimentation is a positive sign that the startup is right for you. The correct definition of a successful experiment is “an experiment that taught us something new”. The common answer of “an experiment that confirmed our preconceived notions” (or in A/B testing terms: “an experiment where we shipped the test variation”) is absolutely wrong.

                                                                                                                                                              Q7: Do leaders at the company explicitly seek truthful data, even when the truth may expose their mistakes? As with Q4, this may be best probed by asking leaders for examples of cases where they uncovered data that proved them wrong. Startups that harbour a culture of hiding from bad news are best avoided by excellent data people. In my experience and based on countless stories by friends, avoidance of bad news and truthful data becomes more common as companies grow. Great startup leaders care about the success of their business and know that hiding from the truth isn’t going to make it disappear.

                                                                                                                                                              Q8: How is uncertainty quantified and communicated? How does it affect decisions? Common sources of uncertainty include sampling biases and missing or wrong data. Marketers are especially notorious for ignoring uncertainty for the sake of memorability (“nine out of ten doctors agree…”). But ignoring uncertainty has long been a way of getting data driven off a cliff. This is at the core of why I recommend the Calling Bullshit book and course to any aspiring data professional. Don’t work with startups that exhibit bullshit failure modes and ignore uncertainty – unless you have the mandate to shape the data culture for the better.

                                                                                                                                                              Data-to-AI health beyond culture

                                                                                                                                                              This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                              You can download a guide containing all the questions as a PDF. The next area I’ll cover is Processes & Project Management – aspects of delivery that are more formal than the somewhat-intangible Culture. Feedback is always welcome!

                                                                                                                                                              Subscribe
                                                                                                                                                                -
                                                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/index.html b/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/index.html index 215dfd847..136ff31d6 100644 --- a/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/index.html +++ b/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/index.html @@ -2,7 +2,7 @@

                                                                                                                                                                Plumbing, Decisions, and Automation: De-hyping Data & AI

                                                                                                                                                                May 27, 2024
                                                                                                                                                                contrasting an amateur and a professional otter; the amateur asks about tools, the professional asks about plumbing, decisions, and automation

                                                                                                                                                                Data & AI health is hard to define. Recently, it occurred to me that its essence can be distilled with three questions:

                                                                                                                                                                1. Plumbing: What’s the state of your data engineering lifecycles?
                                                                                                                                                                2. Decisions: How do you use descriptive, predictive, and causal modelling to support decisions?
                                                                                                                                                                3. Automation: How do you use AI to automate processes?

                                                                                                                                                                These questions help identify gaps and opportunities. While each question focuses on the present state, it’s natural to follow up with plans for a brighter future.

                                                                                                                                                                In practice, you would go deep on each area. Each question is a door that leads to a corridor with many more doors.

                                                                                                                                                                Amateurs versus professionals

                                                                                                                                                                If you’ve ever worked with data, you’d have a sense of what amateur and professional answers to the above questions may look like. In practice, answers are multifaceted and fall on a continuum. But here are some simplified examples from each end of the continuum:

                                                                                                                                                                AmateurProfessional
                                                                                                                                                                PlumbingRudimentary pipelines, manually-populated spreadsheetsAll necessary data is trustworthy and available on tap
                                                                                                                                                                DecisionsRelying on one-off charts and models, along with the intuition of HiPPOs (highest-paid persons’ opinions)Relying on relevant data and modelling efforts that are proportional to the gravity of each decision
                                                                                                                                                                AutomationSuperficial use of off-the-shelf toolsDeep, mindful integration of tech to replace manual work where it delivers the most value

                                                                                                                                                                Going down the rabbit hole

                                                                                                                                                                The three areas pretty much define my career, but there is always much more to learn. The main message of this post is that little has changed since Harrington Emerson uttered these words in 1911:

                                                                                                                                                                As to methods, there may be a million and then some, but principles are few. The person who grasps principles can successfully select their own methods. The person who tries methods, ignoring principles, is sure to have trouble.

                                                                                                                                                                (OK, one thing did change – Emerson used man rather than person, but I fixed it for him.)

                                                                                                                                                                You can explore further with these posts:

                                                                                                                                                                1. Plumbing: Fully understanding the data engineering lifecycle is more important than mastering a single tool.
                                                                                                                                                                2. Decisions: According to my 2018 definition, this is what data science is all about. There’s endless depth to building descriptive, predictive, and causal models. But the key to rising above tool hype is understanding the why of data science, which is to support decisions.
                                                                                                                                                                3. Automation: The term AI is around peak hype right now. This makes it easy for cynics to dismiss the over-excited claims of AI proponents. Avoid cynicism – simply think of AI as automation and understand that relentless but mindful automation is key to success in our world.

                                                                                                                                                                More questions to probe the Data-to-AI health of startups

                                                                                                                                                                This post is a slight detour from the series on my Data-to-AI Health Check for Startups. I figured it’s a valuable detour since I now see the triad of Plumbing, Decisions, and Automation as the essence of Data & AI health for any organisation.

                                                                                                                                                                Previous posts in the series:

                                                                                                                                                                You can download a guide containing all the questions as a PDF. I’m still planning to cover Processes & Project Management next – hopefully I won’t get detoured again. Feedback is always welcome!

                                                                                                                                                                Subscribe
                                                                                                                                                                  -
                                                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/06/03/how-to-avoid-startups-with-poor-development-processes/index.html b/2024/06/03/how-to-avoid-startups-with-poor-development-processes/index.html index 361ae420d..abe7b0a7b 100644 --- a/2024/06/03/how-to-avoid-startups-with-poor-development-processes/index.html +++ b/2024/06/03/how-to-avoid-startups-with-poor-development-processes/index.html @@ -2,7 +2,7 @@

                                                                                                                                                                  How to avoid startups with poor development processes

                                                                                                                                                                  June 3, 2024
                                                                                                                                                                  minimalist image of the phrase RERO: release early, release often

                                                                                                                                                                  Avoid those who release late, release seldom, and don’t listen to their customers.

                                                                                                                                                                  Many founders have never worked at a startup. This may make them oblivious to failure modes that arise from poor development processes. With poor processes, even the most brilliant people are ineffective, as they’re constantly wasting time and fighting fires.

                                                                                                                                                                  You don’t want to join a startup that’s mired in intractable chaos. To avoid such a place, ask questions from the Processes & Project Management section of my Data-to-AI Health Check for Startups. Do this even if you’re a data scientist or a junior engineer – because everyone suffers in an environment with poor processes.

                                                                                                                                                                  This post briefly explains each question, along with expected answers and suggestions for eliciting informative responses.

                                                                                                                                                                  The questions

                                                                                                                                                                  Q1: How often are changes shipped to production? Even in 2024, there are software startups that don’t follow the RERO philosophy: “Release early. Release often. And listen to your customers.” Sporadic releases are often indicative of poor processes. Stay away from such startups, unless you have a mandate to improve things.

                                                                                                                                                                  Q2: How is the impact of new features quantified before and after their release? Another classic quote in the spirit of RERO is the first principle behind the Agile Manifesto: “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.” And you can’t know what’s valuable software if you don’t collect data from your users and meaningfully aggregate it. Helping with this is likely to be your responsibility if you’re the first data hire – ensure that the founders understand that.

                                                                                                                                                                  Q3: What processes and systems are in place to collect qualitative and quantitative feedback from users (both internal and external)? This takes a more holistic view of user feedback and includes internal users. A common failure mode for internal-facing data teams is to spend too much time satisfying low-value requests, e.g., build a dashboard that is barely used. Given the opportunity cost, it’s important to collect truthful feedback from all users.

                                                                                                                                                                  Q4: Were there any outages in recent months? How did they affect users? What changes did you implement to reduce the chance of outages and impact on users? Some outages are unavoidable, especially early on. If the answers reveal that there are repeated issues that go unaddressed, the team may be too busy fighting fires rather than shipping valuable software. This applies to internal systems as well, e.g., data ingestion pipelines that keep breaking shouldn’t be seen as normal.

                                                                                                                                                                  Q5: What system do you use for prioritising and tracking work across the company? This includes project management tools, and their use in practice – but any tool can be misused. The important things to look for are that: (1) a system exists; and (2) the system will improve over time.

                                                                                                                                                                  Q6: How do you balance paying down tech debt and shipping new features? Tech debt is unavoidable, especially in a fast-growing startup. You’re looking for an acknowledgement of this fact, along with an understanding that accruing too much tech debt reduces the ability to ship new features. While the concept comes from software engineering, it also applies to product analytics: if the data is a tangled mess and pipelines are unreliable (i.e., data tech debt is high), then analytics can’t be trusted to help improve the product.

                                                                                                                                                                  Q7: What proportion of the engineering and data staff time is spent on: (1) dealing with bugs and incidents; (2) meetings and admin overheads; and (3) shipping new features? This question takes Q5 & Q6 from philosophy to practice – “don’t tell me your priorities; show me your calendar”. If individual contributors don’t spend most of their time shipping new features (including research into the necessity of these features), there may be too much tech debt or too many overheads.

                                                                                                                                                                  Q8: What are the key team rituals (e.g., recurring meetings, sprint planning, demos, postmortems, standups)? Again, the calendar is a source of insights. For early-stage startups, awareness of the need for rituals may be low, and they may be implemented poorly. For example, I believe that daily synchronous standups are a bad idea for remote teams. A better approach is using a bot that asks everyone for their progress, upcoming tasks, and blockers – and following up to ensure that everyone is on track as a team.

                                                                                                                                                                  Q9: What processes are in place for code, design, and architecture reviews? The depth and time of each process should be proportional to the magnitude and impact of the change. For example, a trivial code improvement in a product that barely has any users can be shipped with a post-commit review. In contrast, migrating a live product to a different database system requires deeper consultation.

                                                                                                                                                                  Q10: Are there different processes for internal-facing data products (e.g., custom admin dashboards)? The unfortunate reality of many organisations is that data analytics and software engineering live in different silos. This can easily happen at small startups as well, with a single analyst working in isolation from product teams. I believe that the ideal situation is that internal-facing data products are treated like the software products that they are. They may not need to be as pretty as external-facing products, but their development should follow processes that ensure high quality, trustworthiness, and satisfaction of user needs.

                                                                                                                                                                  Q11: Give an example of how the above items manifest in a big project that was recently completed. It’s often hard to speak of abstract processes. An example of a big project may be the best way of moving from the abstract to the everyday reality of the startup.

                                                                                                                                                                  Q12: Are there any gaps in the current processes or changes you’d like to introduce? This probes for a growth mindset. If the answer is no, there’s probably something wrong.

                                                                                                                                                                  What if you can’t fit in all the questions?

                                                                                                                                                                  If you’re simply trying to avoid a dysfunctional startup rather than dive deep into development processes, a subset of the questions will suffice:

                                                                                                                                                                  • Q1 + Q2: Ensure that products are continuously improving based on user feedback.
                                                                                                                                                                  • Q7: Ensure that time spent by current staff aligns with how you want to spend your time.
                                                                                                                                                                  • Q11 + Q12: Ensure that leaders are conscious of the need to implement and improve processes.

                                                                                                                                                                  Data-to-AI health beyond processes

                                                                                                                                                                  This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                                  You can download a guide containing all the questions as a PDF. The next area of the health check is Data (finally!). Feedback is always welcome!

                                                                                                                                                                  Subscribe
                                                                                                                                                                    -
                                                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/index.html b/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/index.html index 19a890798..5feca2972 100644 --- a/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/index.html +++ b/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/index.html @@ -13,7 +13,7 @@

                                                                                                                                                                    Burmistrov’s post is worth reading, as it delves deeper into specific examples of how such events are useful for observability and exploration of unknown unknowns: things you weren’t aware would be interesting when the code emitting the event was written.

                                                                                                                                                                    Having been on both the producing and consuming side of such event streams, I have a few thoughts to add:

                                                                                                                                                                    • A common objection to Burmistrov’s post is the cost of tracking. This depends on implementation specifics – there are ways to balance cost and usefulness of events. Further, if a startup is growing and more events are emitted, revenue and funding should also grow – which supports higher-volume tracking. Further, tracking costs should be compared to the opportunity cost of data gaps and to the cost of engineering time for cost-optimising the tracking system.
                                                                                                                                                                    • Many software engineers don’t think in terms of events, as data models for production systems typically reflect the current state of the product rather than its entire history. This is unlikely to radically change, as this sort of data model makes sense in production. To bring engineers along for the event ride, it’s worth getting them to read the classic article about The Log by Jay Kreps.
                                                                                                                                                                    • As a step towards data quality assurance by event producers, automated tests should ensure that events are emitted as expected. This is different from traditional logging, which is seen as a side effect of the system that doesn’t require testing.

                                                                                                                                                                    What are Kukuyeva’s five business aspects?

                                                                                                                                                                    I came across Irina Kukuyeva last year, when I was looking for other people who are offering Chief Data Officer engagements. Her website is a treasure trove of advice for founders, investors, data consultants, and others.

                                                                                                                                                                    On the topic of data that should be tracked, she has this to say:

                                                                                                                                                                    After 10 years of collaborating with companies of all shapes, sizes and industries, I’ve found that all aspects of your business fall into just 5 attributes that you should (legally and ethically) track across your platform/hardware/service and your customers, timestamped at the individual event level:

                                                                                                                                                                    1. “Demographics” of the customer’s/IoT device (e.g., Android/iPhone, tablet, desktop, hardware type),
                                                                                                                                                                    2. Your app’s/platform’s “demographics” (e.g., release 0.9.3, pricing plan(s)),
                                                                                                                                                                    3. State(s) of the app’s/platform’s assets (e.g., inventory, sensor(s)),
                                                                                                                                                                    4. Interactions of all of the above, and
                                                                                                                                                                    5. Touch-points (e.g. acquisition channels, marketing emails, customer service interactions, sensor maintenance)

                                                                                                                                                                    One thing I found confusing when I first read the article was the use of the word attributes, which commonly refers to specific event attributes. However, now I understand that not all aspects are tracked in each event – it’s a broad overview of what should ideally be tracked across the business.

                                                                                                                                                                    Importantly, when thinking of tracking at the logical level, the tools don’t matter. This is exemplified by another Kukuyeva article that advises founders to get started on their data strategy with tools that can be as simple as pen and paper.

                                                                                                                                                                    For examples of how events fold up to analytics and other use cases, see Eventify Everything by Timo Dechau and Activity Schema. However, keep in mind this observation by Misha Panko of Motif Analytics:

                                                                                                                                                                    It took the world decades to develop widely accepted standards for working with relational data and SQL. I believe we are at the early stages of doing the same with event data and sequence analytics. It is starting to simultaneously emerge in many different fields:

                                                                                                                                                                    • eng observability (traces at Datadog, Sumologic, etc)
                                                                                                                                                                    • operational research (process mining at Celonis)
                                                                                                                                                                    • product analytics (funnels at Amplitude, Mixpanel)

                                                                                                                                                                    As with every new field, there are a lot of different and overlapping terms being suggested and explored at the same time.

                                                                                                                                                                    In short, even though systems for event stream processing like Apache Kafka are now over a decade old, the industry is still figuring out how to best model and use all the event data. No one has all the answers, and your mileage will vary.

                                                                                                                                                                    What’s a healthy level of tracking?

                                                                                                                                                                    Assuming compliance with relevant data privacy regulations, there’s still a question of what tracking level is healthy. Despite the yes/no phrasing of my original question (“do you track Kukuyeva’s five business aspects as wide events?”), responses are likely to fall on a continuum between 0 and 1:

                                                                                                                                                                    1. Obviously unhealthy: No, we don’t track any events.
                                                                                                                                                                    2. Unrealistically healthy: Yes, we track everything and have high confidence that we know everything we need to know.

                                                                                                                                                                    Health is relative to the stage of the startup and its current goals. If data gaps are blocking growth or likely to start hurting growth in the 6-12 month horizon, then tracking is insufficiently healthy and should be addressed.

                                                                                                                                                                    Data-to-AI health beyond event tracking

                                                                                                                                                                    This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                                    You can download a guide containing all the questions as a PDF. Next, I’ll go into the other questions in the Data section. Feedback is always welcome!

                                                                                                                                                                    Subscribe
                                                                                                                                                                      -
                                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/06/17/ai-aint-gonna-save-you-from-bad-data/index.html b/2024/06/17/ai-aint-gonna-save-you-from-bad-data/index.html index ec243da8d..ddbec92e8 100644 --- a/2024/06/17/ai-aint-gonna-save-you-from-bad-data/index.html +++ b/2024/06/17/ai-aint-gonna-save-you-from-bad-data/index.html @@ -2,7 +2,7 @@

                                                                                                                                                                      AI ain't gonna save you from bad data

                                                                                                                                                                      June 17, 2024
                                                                                                                                                                      bad data monster with a helpless robot

                                                                                                                                                                      Now that we have generative AI, we no longer need to worry about data, right? Well, we’re not quite there yet. On their own, ChatGPT, Gemini, and their friends can’t save us from bad decisions around data collection and modelling, or from poorly-designed metrics.

                                                                                                                                                                      While we wait for better AI agents to replace data scientists and engineers, I propose we ask a standard set of six questions about the data health of any project. These questions come from my Data-to-AI Health Check for Startups, but they apply anywhere. You can use them as a starting point when joining a new initiative, or to assess the state of an existing project.

                                                                                                                                                                      Before we start, note that the goal is to identify gaps as opportunities for improvement. It’s easy to see data issues as insurmountable, but despair isn’t a viable data strategy. Aim to adopt Stockdale-style optimism when dealing with data:

                                                                                                                                                                      You must never confuse faith that you will prevail in the end – which you can never afford to lose – with the discipline to confront the most brutal facts of your current reality, whatever they might be.

                                                                                                                                                                      As with other posts in the health check series, this post provides a brief explanation for every question.

                                                                                                                                                                      Let’s jump in.

                                                                                                                                                                      The questions

                                                                                                                                                                      Q1: Do you track Kukuyeva’s five business aspects as wide events?

                                                                                                                                                                      This question is foundational, as inadequate instrumentation often makes data-informed decisions impossible. Given its importance, I wrote a separate post on this question.

                                                                                                                                                                      The short story is that you need event-based tracking and event data modelling of key aspects of the business and product. Events are essentially timestamped mappings. For example, a customer purchase is an event that should be logged along with metadata on the customer and platform at the time of purchase.

                                                                                                                                                                      Q2: Is there data you need that isn’t collected or is inaccessible? What is stopping you from obtaining it?

                                                                                                                                                                      While Q1 covers high-level event tracking, Q2 brings it down to specific needs.

                                                                                                                                                                      Sometimes, data can’t be collected due to practical or legal reasons. In some cases, other data can serve as a proxy. For example, companies are typically interested in measuring “customer satisfaction”, but asking directly about satisfaction suffers from a host of problems (e.g., not everyone responds, and the timing and phrasing of questions influence answers). Instead, customer satisfaction can be partly inferred from behaviour like repeat purchases – but that’s also not perfect.

                                                                                                                                                                      In any case, seeking perfection in data is a recipe for disappointment. You should start with business needs and find the data to best address them within a reasonable timeframe.

                                                                                                                                                                      Q3: On a scale of 1 to 5, rate the quality of your key datasets. If you’re unsure due to limited observability and quality checks, it’s a 1.

                                                                                                                                                                      This question should be answered by those who are closest to the data: usually data engineers, scientists, analysts, or one of the dozens of other titles data specialists go by. An experienced data specialist would have a subjective sense of data quality, so it’s worth agreeing on quality definitions if the rating is done by multiple people. Generally, a dataset is of high quality if it’s fit for its intended uses in decisions and automation.

                                                                                                                                                                      Again, absolute perfection is impossible: Data is a model of the world, and all models are wrong, but some are useful.

                                                                                                                                                                      Q4: On a scale of 1 to 5, what is the confidence of stakeholders in the data and metrics that are used to make business decisions? Explain why.

                                                                                                                                                                      Low data quality often leads to low trust and confidence in the outputs of data specialists. However, that’s not always the case. Sometimes, stakeholders may have high confidence in metrics because they’re unaware of underlying data issues. In other cases, confidence is low due to historical reasons: Trust takes time to build – it is a trailing indicator of consistently making and keeping promises. Data specialists are sometimes enamoured with fancy tech and tools, neglecting simple wins that are at the foundation of data’s hierarchy of needs. After decades of hype (from big data through data science to AI agents), I can see why many people treat anything that falls under Data & AI with suspicion. If you’re a data specialist, serving your customers with relevant trustworthy data can be a rare delight.

                                                                                                                                                                      Q5: If you are currently using advanced AI/ML, do you have all the data you need for the models to perform as accurately as required?

                                                                                                                                                                      By advanced AI/ML, I mean fine-tuning or building machine learning models from scratch. This is distinct from basic AI/ML, which relies on third-party models as black boxes. For example, calling a vision API to extract text from images is basic AI/ML. Training a model on your proprietary image data is advanced AI/ML. The latter requires data of sufficient quality for model accuracy, where sufficient accuracy depends on the application.

                                                                                                                                                                      Implicit in this question are satisfactory answers to Q1-Q3: You need to be tracking advanced AI/ML performance in the context where it’s used (Q1), have access to the data you need (Q2), and ensure that data quality is fit for advanced AI/ML (Q3). Advanced AI/ML is hard to do well, and failure can erode stakeholder trust (Q4). However, “failure” depends on expectations – AI/ML models are probabilistic, so setting the right expectations is key. For example, as ChatGPT has shown, it’s possible to build a useful consumer product on top of an AI/ML model that is often wrong.

                                                                                                                                                                      Q6: If you are planning new advanced AI/ML projects, do you have all the data you need for them? If not, what is the effort required to obtain the data? Is it time-sensitive (e.g., ingesting a public dataset is less time-sensitive than starting to collect timestamped proprietary data)?

                                                                                                                                                                      This is the future-oriented version of Q5. It’s best to think of data and metrics before kicking off advanced AI/ML projects. Further, it’s better to start without AI/ML than over-complicate things early on.

                                                                                                                                                                      In short, user needs should inform project decisions on what to build. These decisions and plans then inform data collection. Don’t make the common mistake of starting with shiny tech as the proverbial hammer that’s looking for nail-shaped problems.

                                                                                                                                                                      Don’t forget the opportunities!

                                                                                                                                                                      Data is much like solar energy: It exists even if you don’t capture it, and most of it bounces back to space unused. Harnessed wisely, it can power business decisions and become a differentiator for your product.

                                                                                                                                                                      However, when working closely with data, it’s easy to feel despair due to the never-ending stream of quality issues and stakeholder requests. I’ve felt this despair myself many times.

                                                                                                                                                                      For me, the cure for data despair comes from shifting focus from gaps to opportunities:

                                                                                                                                                                      1. Map the current state of data, including key gaps.
                                                                                                                                                                      2. Learn about relevant business opportunities from internal/external customers and industry peers.
                                                                                                                                                                      3. Create a plan to incrementally improve the state of data, and seize opportunities starting with the lowest-hanging fruit.
                                                                                                                                                                      4. Execute the plan.
                                                                                                                                                                      5. Repeat steps 1-4 periodically.

                                                                                                                                                                      Data-to-AI health beyond abstract data

                                                                                                                                                                      This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                                      You can download a guide containing all the questions as a PDF. Next, I’ll go into the questions from the Tech section, which are directly related to how the abstract Data questions manifest in practice. Feedback is always welcome!

                                                                                                                                                                      Subscribe
                                                                                                                                                                        -
                                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/index.html b/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/index.html index 540e4a7b3..969d05c65 100644 --- a/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/index.html +++ b/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/index.html @@ -2,7 +2,7 @@

                                                                                                                                                                        Is your tech stack ready for data-intensive applications?

                                                                                                                                                                        June 24, 2024
                                                                                                                                                                        a stack of computers, wires, and hay in an office area

                                                                                                                                                                        Data-intensive projects fail when you treat them like traditional software projects. But they also fail when you don’t apply best practices from software engineering.

                                                                                                                                                                        Why?

                                                                                                                                                                        Because data-intensive systems are made of data, and also made of software. Therefore:

                                                                                                                                                                        1. data changes can lead to failures; and
                                                                                                                                                                        2. software changes can lead to failures.

                                                                                                                                                                        In traditional software systems, you fully control the changes. Your software doesn’t change unexpectedly.

                                                                                                                                                                        In data-intensive systems, you cede control to the data. The data changes constantly, and it affects the behaviour of your system.

                                                                                                                                                                        To succeed, you need to manage both the data and software aspects of your systems. This successful management is the essence of the questions from the Tech section of my Data-to-AI Health Check for Startups. This post presents the questions along with guidance on what constitutes healthy answers.

                                                                                                                                                                        What do I mean by data intensity?

                                                                                                                                                                        For the last few months, I have set my LinkedIn tagline to “helping startups ship data-intensive solutions (AI/ML for climate/nature tech)”. I landed on it after a bit of a struggle with succinctly defining exactly what it is I do.

                                                                                                                                                                        The problem is that after over a decade of “data” roles, I don’t see the field of AI/ML (artificial intelligence and machine learning) as a sanctified sphere that’s separate from real-world data and humans. Further, while business intelligence (aka analytics) is seen by some as less “sexy” than AI/ML, I see it as a different lens of using data to drive business outcomes. Essentially, it all comes down to plumbing, decisions, and automation.

                                                                                                                                                                        In the days of the Big Data hype, much attention was given to the three Vs of data: Volume, Velocity, and Variety – what flows through the plumbing. To me, data intensity goes beyond the three Vs. This is how I define it in the first section of my Data-to-AI Health Check:

                                                                                                                                                                        High data intensity typically requires low-latency processing of large volumes of data with more than one database server. With high intensity, data processing issues noticeably affect key business metrics.

                                                                                                                                                                        That is, in data-intensive settings, data issues affect decisions and automation in a way that hurts the business.

                                                                                                                                                                        A couple of examples may help:

                                                                                                                                                                        • Low intensity: A dashboard that doesn’t contain any actionable metrics. If the metrics change due to bugs in the data processing, it doesn’t affect decisions.
                                                                                                                                                                        • High intensity: An ad-serving platform that personalises ads in real time based on numerous data points. If any model or system breaks, millions of dollars may be lost.

                                                                                                                                                                        In short, the higher the data intensity, the more the flow of data affects the bottom line.

                                                                                                                                                                        Understanding tech stacks and lifecycles

                                                                                                                                                                        At 15 questions, the Tech section of my Data-to-AI Health Check for Startups is long and deep. To keep this post digestible, I won’t go into every question. Instead, I’ve grouped the questions by theme.

                                                                                                                                                                        First up, on the tech stacks and lifecycles:

                                                                                                                                                                        • Q1: Provide an architecture diagram for your tech systems (product and data stacks), including first-party and third-party tools and databases. If a diagram doesn’t exist, an ad hoc drawing would work as well.
                                                                                                                                                                        • Q2: Zooming in on data stacks, what tools and pipelines do you use for the data engineering lifecycles (generation, storage, ingestion, transformation, and serving), and downstream uses (analytics, AI/ML, and reverse ETL)?
                                                                                                                                                                        • Q3: Zooming in further on the downstream uses of analytics and AI/ML, what systems, processes, and tools do you use to manage their lifecycles (discovery, data preparation, model engineering, deployment, monitoring, and maintenance)? Give specific project examples.
                                                                                                                                                                        • Q4: Are there any tech choices you regret? Why?
                                                                                                                                                                        • Q5: Are there any new tools you want to introduce to your stack? Why?

                                                                                                                                                                        To an extent, tech stacks and lifecycles follow the Anna Karenina principle: All healthy stacks are alike; each unhealthy stack is unhealthy in its own way.

                                                                                                                                                                        By asking for their descriptions, I’m aiming to uncover gaps and opportunities.

                                                                                                                                                                        Often, some gaps are known to the people in charge, but they haven’t been explicitly discussed. This is especially common in startups, where competing priorities and resource constraints require compromising on scope and quality to fuel growth. In addition, it’s impossible for small startups to have all the relevant experts on the founding team, so best practices aren’t followed due to ignorance rather than due to intentional compromises made to move fast. However, a lack of awareness of best practices can often lead to the startup moving too slowly.

                                                                                                                                                                        Two concrete examples:

                                                                                                                                                                        • many people outside the data world are unaware of recent advances in tooling for management of data transformations (dbt and its competitors), and
                                                                                                                                                                        • practitioners who’ve only built ML models in academia rarely appreciate the complexity of running ML in production (MLOps is much more than ML).

                                                                                                                                                                        Beyond gaps that may be exposed by Q1-Q3, explicitly asking about regrettable and future tech choices (Q4 & Q5) helps surface evidence of an overreliance on unproven or exotic tech (aka wasted innovation tokens) and an underreliance on proven tech (aka reinvention of wheels). This is especially common with inexperienced operators who are too excited about playing with shiny tools. Use of unproven tech should be reserved to the cases where it confers a competitive advantage (e.g., being first to market with the latest AI advances).

                                                                                                                                                                        Basic quality assurance and delivery

                                                                                                                                                                        The next set of questions covers what I consider to be the basics of quality assurance and continuous delivery:

                                                                                                                                                                        • Q6: How do you test product code and infrastructure setup? How good is the coverage (formally – percentage of statements covered, and conceptually – confidence from 1 to 5 that tests capture faults prior to deployment)?
                                                                                                                                                                        • Q7: Do all tests run automatically on every version of the code?
                                                                                                                                                                        • Q8: Are deployments done as a single automated step (e.g., push new containers to production when the main branch is updated)?
                                                                                                                                                                        • Q9: How faithful are development, testing, and staging environments to the production setup? Are there gaps that can be feasibly addressed? If so, what is stopping you from addressing them?

                                                                                                                                                                        As I’m writing this in 2024, all the tooling exists to set things up with solid testing and deployment processes – and it’s constantly getting easier. The only place where such processes may be skipped is in throwaway prototypes, where testing unnecessarily slows things down.

                                                                                                                                                                        Being a startup is also not an excuse. As Martin Fowler pointed out years ago, the internal quality of software doesn’t incur a cost. That is, by implementing solid systems and processes for automated testing and deployment, teams move faster. Teams that cut corners on internal quality may move faster in the very short term, but typically get overtaken by their higher-internal-quality counterparts within weeks.

                                                                                                                                                                        No startup aims to be around only for a few weeks, so investing in internal quality is key to tech health.

                                                                                                                                                                        In Fowler’s words:

                                                                                                                                                                        • Neglecting internal quality leads to rapid build up of cruft
                                                                                                                                                                        • This cruft slows down feature development
                                                                                                                                                                        • Even a great team produces cruft, but by keeping internal quality high, is able to keep it under control
                                                                                                                                                                        • High internal quality keeps cruft to a minimum, allowing a team to add features with less effort, time, and cost

                                                                                                                                                                        Unfortunately, some software engineers never learn this lesson. Further, data professionals that don’t have a software background are even less likely to be exposed to the importance of internal quality and how it can be enforced.

                                                                                                                                                                        That said, it’s never too late to learn and improve. This is key to avoiding failure modes that arise in data projects when best practices from software engineering aren’t applied.

                                                                                                                                                                        Specific data-intensive failure modes

                                                                                                                                                                        The next set of questions probes for failure modes that are specific to data-intensive work (data engineering, analytics, AI/ML, etc.):

                                                                                                                                                                        • Q10: Do you apply the same standards of testing and deploying product code to data? For example, is there untested SQL code hidden in dashboarding tools or the database layer, or is SQL treated like core product code (tracked in source control with isolated testing)?
                                                                                                                                                                        • Q11: How are schema changes managed and tested in each data system?
                                                                                                                                                                        • Q12: Do you rely on notebooks for production data code? If so, how do you ensure that notebook code meets the same quality standards as core product code (especially around testing and change management)?
                                                                                                                                                                        • Q13: Do advanced AI/ML projects meet your performance expectations? If not, do you know how to improve performance without data changes?

                                                                                                                                                                        Data-intensive work is essentially about building models with software:

                                                                                                                                                                        • Raw data is a model of real-world entities and events, expressed in database schemas (even “schemaless” databases have a schema – it’s just unbounded).
                                                                                                                                                                        • Dashboards present models of metrics that originate in raw data, with the goal of informing decisions.
                                                                                                                                                                        • AI/ML models are essentially complex data transformations, e.g., from a matrix of pixels to a probability that the image modelled by the pixels is of a cat or a dog.

                                                                                                                                                                        Due to historical and practical reasons, much of this modelling work is done by people with no training in software engineering. While the industry is maturing, Q10-13 often expose gaps. The ideal answer to each question is that all models are fully tested and managed – just like software, but with extra care for the complexity introduced by data.

                                                                                                                                                                        Maintaining long-term success

                                                                                                                                                                        Finally, the last two questions cover monitoring and maintenance:

                                                                                                                                                                        • Q14: On a scale of 1 to 5, how confident are you in detecting and addressing issues in production (including product, infra, data, and ML observability 1.0 & 2.0)? Do you have action plans to increase your level of confidence?
                                                                                                                                                                        • Q15: What DevOps, DataOps, and MLOps practices do you follow that weren’t covered above? Are there known gaps and plans to address them?

                                                                                                                                                                        Even if a data-intensive project is considered “done”, it still changes in production due to its dependence on data. The degree of likely change varies by project, but it needs to be actively managed for long-term success.

                                                                                                                                                                        Data-to-AI health beyond the tech

                                                                                                                                                                        This post is part of a series on my Data-to-AI Health Check for Startups. Previous posts:

                                                                                                                                                                        You can download a guide containing all the questions as a PDF. Next, I’ll go into the questions from the Security & Compliance section. Feedback is always welcome!

                                                                                                                                                                        Subscribe
                                                                                                                                                                          -
                                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/causal-inference-resources/index.html b/causal-inference-resources/index.html index c0d8bb89a..3d75e7a84 100644 --- a/causal-inference-resources/index.html +++ b/causal-inference-resources/index.html @@ -8,7 +8,7 @@ Causal Inference: What if by Miguel Hernán and Jamie Robins: The most practical book I’ve read. Highly recommended. Trustworthy Online Controlled Experiments : A Practical Guide to A/B Testing by Ron Kohavi, Diane Tang, and Ya Xu: Building on the authors’ decades of industry experience, this is pretty much the bible of online experiments, which is how causal inference is often done in practice.">

                                                                                                                                                                          Causal inference resources

                                                                                                                                                                          This is a list of some causal inference resources, which I update from time to time. You can also check out my posts on causal inference and A/B testing.

                                                                                                                                                                          Books:

                                                                                                                                                                          Articles:

                                                                                                                                                                          Courses:

                                                                                                                                                                            Subscribe
                                                                                                                                                                              -
                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/contact/index.html b/contact/index.html index a318d1f09..e6f3e7fe4 100644 --- a/contact/index.html +++ b/contact/index.html @@ -1,7 +1,7 @@ Stay in touch | Yanir Seroussi | Data & AI for Startup Impact -

                                                                                                                                                                              Stay in touch

                                                                                                                                                                              Logo of Yanir Seroussi's consulting services, depicting a wave and an up-and-to-the-right graph next to his profile picture.

                                                                                                                                                                              Contact me

                                                                                                                                                                              Feel free to contact me about topics discussed on this website, potential work, or anything else you think I’d find interesting. Contact options:

                                                                                                                                                                              Subscribe to my mailing list

                                                                                                                                                                              To get new posts delivered to your mailbox, enter your email address below and tap Subscribe. You may unsubscribe at any time.

                                                                                                                                                                                +

                                                                                                                                                                                Stay in touch

                                                                                                                                                                                Logo of Yanir Seroussi's consulting services, depicting a wave and an up-and-to-the-right graph next to his profile picture.

                                                                                                                                                                                Contact me

                                                                                                                                                                                Feel free to contact me about topics discussed on this website, potential work, or anything else you think I’d find interesting. Contact options:

                                                                                                                                                                                Subscribe to my mailing list

                                                                                                                                                                                To get new posts delivered to your mailbox, enter your email address below and tap Subscribe. You may unsubscribe at any time.

                                                                                                                                                                                  -
                                                                                                                                                                                  Alternatively, subscribe to RSS feed.
                                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | +
                                                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.
                                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & PaperMod
                                                                                                                                                                                      \ No newline at end of file diff --git a/data-to-ai-health-check/index.html b/data-to-ai-health-check/index.html index 1a5d7b70c..9f5f33470 100644 --- a/data-to-ai-health-check/index.html +++ b/data-to-ai-health-check/index.html @@ -1,5 +1,5 @@ Free Guide: Data-to-AI Health Check for Startups | Yanir Seroussi | Data & AI for Startup Impact -

                                                                                                                                                                                      Free Guide: Data-to-AI Health Check for Startups

                                                                                                                                                                                      Cover page of the free guide: Data-to-AI Health Check for Startups.

                                                                                                                                                                                      Are you…

                                                                                                                                                                                      • …a startup leader considering your first AI/ML project?
                                                                                                                                                                                      • …a data professional thinking of joining a startup?
                                                                                                                                                                                      • …an investor assessing the viability of a startup’s AI/ML plans?
                                                                                                                                                                                      • …a consultant helping startups ship data-intensive solutions?

                                                                                                                                                                                      If the answer is yes to any of the above, download my free Data-to-AI Health Check for Startups PDF to help guide your next steps.

                                                                                                                                                                                      Startup failure rates are high.
                                                                                                                                                                                      Data/AI/ML projects often fail to ship.
                                                                                                                                                                                      Increase your chances of success with the Data-to-AI Health Check for Startups.

                                                                                                                                                                                        +

                                                                                                                                                                                        Free Guide: Data-to-AI Health Check for Startups

                                                                                                                                                                                        Cover page of the free guide: Data-to-AI Health Check for Startups.

                                                                                                                                                                                        Are you…

                                                                                                                                                                                        • …a startup leader considering your first AI/ML project?
                                                                                                                                                                                        • …a data professional thinking of joining a startup?
                                                                                                                                                                                        • …an investor assessing the viability of a startup’s AI/ML plans?
                                                                                                                                                                                        • …a consultant helping startups ship data-intensive solutions?

                                                                                                                                                                                        If the answer is yes to any of the above, download my free Data-to-AI Health Check for Startups PDF to help guide your next steps.

                                                                                                                                                                                        Startup failure rates are high.
                                                                                                                                                                                        Data/AI/ML projects often fail to ship.
                                                                                                                                                                                        Increase your chances of success with the Data-to-AI Health Check for Startups.

                                                                                                                                                                                          Early feedback from startup gurus

                                                                                                                                                                                          "A symptom of hard-won personal experience ... a solid list."
                                                                                                                                                                                          "Excellent advice for anybody looking to join an early stage startup."
                                                                                                                                                                                          "Great questions to ask any entrepreneur!"

                                                                                                                                                                                          In the guide, you’ll find a set of questions to help you assess a startup’s data/AI/ML health, along with the business context in which data/AI/ML projects get shipped. The questions and scoring guidance cover eight areas, resulting in the scorecard shown below.

                                                                                                                                                                                          Deep learning resources

                                                                                                                                                                                          July 6, 2015

                                                                                                                                                                                          This page summarises the deep learning resources I’ve consulted in my album cover classification project.

                                                                                                                                                                                          Tutorials and blog posts

                                                                                                                                                                                          Papers

                                                                                                                                                                                            Subscribe
                                                                                                                                                                                              -
                                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/free-intro-call/index.html b/free-intro-call/index.html index 9ff38dd39..a55ac71d2 100644 --- a/free-intro-call/index.html +++ b/free-intro-call/index.html @@ -1,5 +1,5 @@ Book a free fifteen-minute call | Yanir Seroussi | Data & AI for Startup Impact -

                                                                                                                                                                                              Book a free fifteen-minute call

                                                                                                                                                                                              Yanir Seroussi having a call with a mystery person

                                                                                                                                                                                              Hello! 🐳

                                                                                                                                                                                              So… you’ve been poking around my website and LinkedIn profile, and you think we’d benefit from a quick intro call.

                                                                                                                                                                                              Well, you’ve come to the right place!

                                                                                                                                                                                              But first, note that there are other ways to contact me that may be more effective.

                                                                                                                                                                                              Second, I also offer longer one-hour advisory calls. You get to set the agenda for such calls, but the focus there is typically on your Data-to-AI Strategy.

                                                                                                                                                                                              Third, nothing is truly free. Filling the form below will subscribe you to my weekly mailing list. If you made it here, you probably want that anyway (you can always unsubscribe).

                                                                                                                                                                                              Once you submit the form, you’ll receive a confirmation email with the booking link for the call.

                                                                                                                                                                                                +

                                                                                                                                                                                                Book a free fifteen-minute call

                                                                                                                                                                                                Yanir Seroussi having a call with a mystery person

                                                                                                                                                                                                Hello! 🐳

                                                                                                                                                                                                So… you’ve been poking around my website and LinkedIn profile, and you think we’d benefit from a quick intro call.

                                                                                                                                                                                                Well, you’ve come to the right place!

                                                                                                                                                                                                But first, note that there are other ways to contact me that may be more effective.

                                                                                                                                                                                                Second, I also offer longer one-hour advisory calls. You get to set the agenda for such calls, but the focus there is typically on your Data-to-AI Strategy.

                                                                                                                                                                                                Third, nothing is truly free. Filling the form below will subscribe you to my weekly mailing list. If you made it here, you probably want that anyway (you can always unsubscribe).

                                                                                                                                                                                                Once you submit the form, you’ll receive a confirmation email with the booking link for the call.

                                                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/kaggle/index.html b/kaggle/index.html index 444624cc1..e4fc707a9 100644 --- a/kaggle/index.html +++ b/kaggle/index.html @@ -2,7 +2,7 @@

                                                                                                                                                                                                    Kaggle competition tips and summaries

                                                                                                                                                                                                    April 5, 2014

                                                                                                                                                                                                    Over the years, I’ve participated in a few Kaggle competitions and wrote a bit about my experiences. This page contains pointers to all my posts, and will be updated if/when I participate in more competitions.

                                                                                                                                                                                                    General advice posts

                                                                                                                                                                                                    Solution posts

                                                                                                                                                                                                    Subscribe
                                                                                                                                                                                                      -
                                                                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/posts/index.html b/posts/index.html index a23ab9042..f31177e64 100644 --- a/posts/index.html +++ b/posts/index.html @@ -11,7 +11,7 @@ ">

                                                                                                                                                                                                      Browse Posts

                                                                                                                                                                                                      Browse my main posts in reverse chronological order below, or by tag. You may also want to check out my shorter-form TIL (today I learned) posts.
                                                                                                                                                                                                        -
                                                                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                        Is your tech stack ready for data-intensive applications?

                                                                                                                                                                                                        Questions to assess the quality of tech stacks and lifecycles, with a focus on artificial intelligence, machine learning, and analytics.

                                                                                                                                                                                                        June 24, 2024

                                                                                                                                                                                                        AI ain't gonna save you from bad data

                                                                                                                                                                                                        Since we’re far from a utopia where data issues are fully handled by AI, this post presents six questions humans can use to assess data projects.

                                                                                                                                                                                                        June 17, 2024

                                                                                                                                                                                                        Startup data health starts with healthy event tracking

                                                                                                                                                                                                        Expanding on the startup health check question of tracking Kukuyeva’s five business aspects as wide events.

                                                                                                                                                                                                        June 10, 2024

                                                                                                                                                                                                        How to avoid startups with poor development processes

                                                                                                                                                                                                        Questions that prospective data specialists and engineers should ask about development processes before accepting a startup role.

                                                                                                                                                                                                        June 3, 2024

                                                                                                                                                                                                        Plumbing, Decisions, and Automation: De-hyping Data & AI

                                                                                                                                                                                                        Three essential questions to understand where an organisation stands when it comes to Data & AI (with zero hype).

                                                                                                                                                                                                        May 27, 2024

                                                                                                                                                                                                        Question startup culture before accepting a data-to-AI role

                                                                                                                                                                                                        Eight questions that prospective data-to-AI employees should ask about a startup’s work and data culture.

                                                                                                                                                                                                        May 20, 2024

                                                                                                                                                                                                        Probing the People aspects of an early-stage startup

                                                                                                                                                                                                        Ten questions that prospective employees should ask about a startup’s team, especially for data-centric roles.

                                                                                                                                                                                                        May 13, 2024

                                                                                                                                                                                                        Business questions to ask before taking a startup data role

                                                                                                                                                                                                        Fourteen questions that prospective employees should ask about a startup’s business model and product, especially for data-focused roles.

                                                                                                                                                                                                        May 6, 2024

                                                                                                                                                                                                        Mentorship and the art of actionable advice

                                                                                                                                                                                                        Reflections on what it takes to package expertise and deliver timely, actionable advice outside the context of employee relationships.

                                                                                                                                                                                                        April 29, 2024

                                                                                                                                                                                                        Assessing a startup's data-to-AI health

                                                                                                                                                                                                        Reviewing the areas that should be assessed to determine a startup’s opportunities and challenges on the data/AI/ML front.

                                                                                                                                                                                                        April 22, 2024

                                                                                                                                                                                                        AI does not obviate the need for testing and observability

                                                                                                                                                                                                        It’s easy to prototype with AI, but production-grade AI apps require even more thorough testing and observability than traditional software.

                                                                                                                                                                                                        April 15, 2024

                                                                                                                                                                                                        My experience as a Data Tech Lead with Work on Climate

                                                                                                                                                                                                        The story of how I joined Work on Climate as a volunteer and became its data tech lead, with lessons applied to consulting & fractional work.

                                                                                                                                                                                                        April 8, 2024

                                                                                                                                                                                                        Artificial intelligence, automation, and the art of counting fish

                                                                                                                                                                                                        Discussing the use of AI to automate underwater marine surveys as an example of the uneven distribution of technological advancement.

                                                                                                                                                                                                        April 1, 2024

                                                                                                                                                                                                        Questions to consider when using AI for PDF data extraction

                                                                                                                                                                                                        Discussing considerations that arise when attempting to automate the extraction of structured data from PDFs and similar documents.

                                                                                                                                                                                                        March 11, 2024

                                                                                                                                                                                                        Two types of startup data problems

                                                                                                                                                                                                        Classifying startups as ML-centric or non-ML is a helpful exercise to uncover the data challenges they’re likely to face.

                                                                                                                                                                                                        March 4, 2024

                                                                                                                                                                                                        Avoiding AI complexity: First, write no code

                                                                                                                                                                                                        Two stories of getting AI functionality to production, which demonstrate the risks inherent in custom development versus starting with a no-code approach.

                                                                                                                                                                                                        February 26, 2024

                                                                                                                                                                                                        Building your startup's minimum viable data stack

                                                                                                                                                                                                        First post in a series on building a minimum viable data stack for startups, introducing key definitions, components, and considerations.

                                                                                                                                                                                                        February 19, 2024

                                                                                                                                                                                                        Nudging ChatGPT to invent books you have no time to read

                                                                                                                                                                                                        Getting ChatGPT Plus to elaborate on possible book content and produce a PDF cheatsheet, with the goal of learning about its capabilities.

                                                                                                                                                                                                        February 12, 2024

                                                                                                                                                                                                        Substance over titles: Your first data hire may be a data scientist

                                                                                                                                                                                                        Advice for hiring a startup’s first data person: match skills to business needs, consider contractors, and get help from data people.

                                                                                                                                                                                                        February 5, 2024

                                                                                                                                                                                                        New decade, new tagline: Data & AI for Impact

                                                                                                                                                                                                        Shifting focus to ‘Data & AI for Impact’, with more startup-related content, increased posting frequency, and deeper audience engagement.

                                                                                                                                                                                                        January 19, 2024

                                                                                                                                                                                                        Supporting volunteer monitoring of marine biodiversity with modern web and data tools

                                                                                                                                                                                                        Summarising the work Uri Seroussi and I did to improve Reef Life Survey’s Reef Species of the World app.

                                                                                                                                                                                                        November 29, 2023

                                                                                                                                                                                                        Lessons from reluctant data engineering

                                                                                                                                                                                                        Video and summary of a talk I gave at DataEngBytes Brisbane on what I learned from doing data engineering as part of every data science role I had.

                                                                                                                                                                                                        October 25, 2023

                                                                                                                                                                                                        My rediscovery of quiet writing on the open web

                                                                                                                                                                                                        Reflections on publishing on this website: Writing publicly to share thoughts and documentation beats chasing views and likes.

                                                                                                                                                                                                        August 28, 2023

                                                                                                                                                                                                        Was data science a failure mode of software engineering?

                                                                                                                                                                                                        Yes, data science projects have suffered from classic software engineering mistakes, but the field is maturing with the rise of new engineering roles.

                                                                                                                                                                                                        June 30, 2023

                                                                                                                                                                                                        How hackable are automated coding assessments?

                                                                                                                                                                                                        Exploring the hackability of speed-based coding tests, using CodeSignal’s Industry Coding Framework as a case study.

                                                                                                                                                                                                        May 26, 2023

                                                                                                                                                                                                        Remaining relevant as a small language model

                                                                                                                                                                                                        Bing Chat recently quipped that humans are small language models. Here are some of my thoughts on how we small language models can remain relevant (for now).

                                                                                                                                                                                                        April 21, 2023

                                                                                                                                                                                                        ChatGPT is transformative AI

                                                                                                                                                                                                        My perspective after a week of using ChatGPT: This is a step change in finding distilled information, and it’s only the beginning.

                                                                                                                                                                                                        December 11, 2022

                                                                                                                                                                                                        Causal Machine Learning is off to a good start, despite some issues

                                                                                                                                                                                                        Reviewing the first three chapters of the book Causal Machine Learning by Robert Osazuwa Ness.

                                                                                                                                                                                                        September 12, 2022

                                                                                                                                                                                                        The mission matters: Moving to climate tech as a data scientist

                                                                                                                                                                                                        Discussing my recent career move into climate tech as a way of doing more to help mitigate dangerous climate change.

                                                                                                                                                                                                        June 6, 2022

                                                                                                                                                                                                        Building useful machine learning tools keeps getting easier: A fish ID case study

                                                                                                                                                                                                        Lessons learned building a fish ID web app with fast.ai and Streamlit, in an attempt to reduce my fear of missing out on the latest deep learning developments.

                                                                                                                                                                                                        March 20, 2022

                                                                                                                                                                                                        Analysis strategies in online A/B experiments: Intention-to-treat, per-protocol, and other lessons from clinical trials

                                                                                                                                                                                                        Epidemiologists analyse clinical trials to estimate the intention-to-treat and per-protocol effects. This post applies their strategies to online experiments.

                                                                                                                                                                                                        January 14, 2022

                                                                                                                                                                                                        Use your human brain to avoid artificial intelligence disasters

                                                                                                                                                                                                        Overview of a talk I gave at a deep learning course, focusing on AI ethics as the need for humans to think on the context and consequences of applying AI.

                                                                                                                                                                                                        November 22, 2021

                                                                                                                                                                                                        Migrating from WordPress.com to Hugo on GitHub + Cloudflare

                                                                                                                                                                                                        My reasons for switching from WordPress.com to Hugo on GitHub + Cloudflare, along with a summary of the solution components and migration process.

                                                                                                                                                                                                        November 10, 2021

                                                                                                                                                                                                        My work with Automattic

                                                                                                                                                                                                        Back-dated meta-post that gathers my posts on Automattic blogs into a summary of the work I’ve done with the company.

                                                                                                                                                                                                        October 7, 2021

                                                                                                                                                                                                        Some highlights from 2020

                                                                                                                                                                                                        Sharing remote teamwork insights, my climate & sustainability activism, Reef Life Survey publications, and progress on Automattic’s Experimentation Platform.

                                                                                                                                                                                                        April 5, 2021

                                                                                                                                                                                                        Many is not enough: Counting simulations to bootstrap the right way

                                                                                                                                                                                                        Going deeper into correct testing of different methods for bootstrap estimation of confidence intervals.

                                                                                                                                                                                                        August 24, 2020

                                                                                                                                                                                                        Software commodities are eating interesting data science work

                                                                                                                                                                                                        Being a data scientist can sometimes feel like a race against software commodities that replace interesting work. What can one do to remain relevant?

                                                                                                                                                                                                        January 11, 2020

                                                                                                                                                                                                        A day in the life of a remote data scientist

                                                                                                                                                                                                        Video of a talk I gave on remote data science work at the Data Science Sydney meetup.

                                                                                                                                                                                                        December 11, 2019

                                                                                                                                                                                                        Bootstrapping the right way?

                                                                                                                                                                                                        Video and summary of a talk I gave at YOW! Data on bootstrap estimation of confidence intervals.

                                                                                                                                                                                                        October 6, 2019

                                                                                                                                                                                                        Hackers beware: Bootstrap sampling may be harmful

                                                                                                                                                                                                        Bootstrap sampling has been promoted as an easy way of modelling uncertainty to hackers without much statistical knowledge. But things aren’t that simple.

                                                                                                                                                                                                        January 7, 2019

                                                                                                                                                                                                        The most practical causal inference book I’ve read (is still a draft)

                                                                                                                                                                                                        Causal Inference by Miguel Hernán and Jamie Robins is a must-read for anyone interested in the area.

                                                                                                                                                                                                        December 24, 2018

                                                                                                                                                                                                        Reflections on remote data science work

                                                                                                                                                                                                        Discussing the pluses and minuses of remote work eighteen months after joining Automattic as a data scientist.

                                                                                                                                                                                                        November 3, 2018

                                                                                                                                                                                                        Defining data science in 2018

                                                                                                                                                                                                        Updating my definition of data science to match changes in the field. It is now broader than before, but its ultimate goal is still to support decisions.

                                                                                                                                                                                                        July 22, 2018

                                                                                                                                                                                                        Advice for aspiring data scientists and other FAQs

                                                                                                                                                                                                        Frequently asked questions by visitors to this site, especially around entering the data science field.

                                                                                                                                                                                                        October 15, 2017

                                                                                                                                                                                                        State of Bandcamp Recommender, Late 2017

                                                                                                                                                                                                        Call for BCRecommender maintainers followed by a decision to shut it down, as I don’t have enough time and Bandcamp now offers recommendations.

                                                                                                                                                                                                        September 2, 2017

                                                                                                                                                                                                        My 10-step path to becoming a remote data scientist with Automattic

                                                                                                                                                                                                        I wanted a well-paid data science-y remote job with an established company that offers a good life balance and makes products I care about. I got it eventually.

                                                                                                                                                                                                        July 29, 2017

                                                                                                                                                                                                        Exploring and visualising Reef Life Survey data

                                                                                                                                                                                                        Web tools I built to visualise Reef Life Survey data and assist citizen scientists in underwater visual census work.

                                                                                                                                                                                                        June 3, 2017

                                                                                                                                                                                                        Customer lifetime value and the proliferation of misinformation on the internet

                                                                                                                                                                                                        There’s a lot of misleading content on the estimation of customer lifetime value. Here’s what I learned about doing it well.

                                                                                                                                                                                                        January 8, 2017

                                                                                                                                                                                                        Ask Why! Finding motives, causes, and purpose in data science

                                                                                                                                                                                                        Video and summary of a talk I gave at the Data Science Sydney meetup, about going beyond the what & how of predictive modelling.

                                                                                                                                                                                                        September 19, 2016

                                                                                                                                                                                                        If you don’t pay attention, data can drive you off a cliff

                                                                                                                                                                                                        Seven common mistakes to avoid when working with data, such as ignoring uncertainty and confusing observed and unobserved quantities.

                                                                                                                                                                                                        August 21, 2016

                                                                                                                                                                                                        Is Data Scientist a useless job title?

                                                                                                                                                                                                        It seems like anyone who touches data can call themselves a data scientist, which makes the title useless. The work they do can still be useful, though.

                                                                                                                                                                                                        August 4, 2016

                                                                                                                                                                                                        Making Bayesian A/B testing more accessible

                                                                                                                                                                                                        A web tool I built to interpret A/B test results in a Bayesian way, including prior specification, visualisations, and decision rules.

                                                                                                                                                                                                        June 19, 2016

                                                                                                                                                                                                        Diving deeper into causality: Pearl, Kleinberg, Hill, and untested assumptions

                                                                                                                                                                                                        Discussing the need for untested assumptions and temporality in causal inference. Mostly based on Samantha Kleinberg’s Causality, Probability, and Time.

                                                                                                                                                                                                        May 14, 2016

                                                                                                                                                                                                        The rise of greedy robots

                                                                                                                                                                                                        Is artificial/machine intelligence a future threat? I argue that it’s already here, with greedy robots already dominating our lives.

                                                                                                                                                                                                        March 20, 2016

                                                                                                                                                                                                        Why you should stop worrying about deep learning and deepen your understanding of causality instead

                                                                                                                                                                                                        Causality is often overlooked but is of much higher relevance to most data scientists than deep learning.

                                                                                                                                                                                                        February 14, 2016

                                                                                                                                                                                                        The joys of offline data collection

                                                                                                                                                                                                        Insights on data collection and machine learning from spending a month sailing, diving, and counting fish with Reef Life Survey.

                                                                                                                                                                                                        January 24, 2016

                                                                                                                                                                                                        This holiday season, give me real insights

                                                                                                                                                                                                        Some companies present raw data or information as “insights”. This post surveys some examples, and discusses how they can be turned into real insights.

                                                                                                                                                                                                        December 8, 2015

                                                                                                                                                                                                        The hardest parts of data science

                                                                                                                                                                                                        Defining feasible problems and coming up with reasonable ways of measuring solutions is harder than building accurate models or obtaining clean data.

                                                                                                                                                                                                        November 23, 2015

                                                                                                                                                                                                        Migrating a simple web application from MongoDB to Elasticsearch

                                                                                                                                                                                                        Migrating BCRecommender from MongoDB to Elasticsearch made it possible to offer a richer search experience to users at a similar cost, among other benefits.

                                                                                                                                                                                                        November 4, 2015

                                                                                                                                                                                                        Miscommunicating science: Simplistic models, nutritionism, and the art of storytelling

                                                                                                                                                                                                        Nutritionism is a special case of misinterpretation and miscommunication of scientific results – something many data scientists encounter in their work.

                                                                                                                                                                                                        October 19, 2015

                                                                                                                                                                                                        The wonderful world of recommender systems

                                                                                                                                                                                                        Giving an overview of the field and common paradigms, and debunking five common myths about recommender systems.

                                                                                                                                                                                                        October 2, 2015

                                                                                                                                                                                                        You don’t need a data scientist (yet)

                                                                                                                                                                                                        Hiring data scientists prematurely is wasteful and frustrating. Here are some questions to ask before you hire your first data scientist.

                                                                                                                                                                                                        August 24, 2015

                                                                                                                                                                                                        Goodbye, Parse.com

                                                                                                                                                                                                        Migrating my web apps away from Parse.com due to reliability issues. Self-hosting is a better solution.

                                                                                                                                                                                                        July 31, 2015

                                                                                                                                                                                                        Learning about deep learning through album cover classification

                                                                                                                                                                                                        Progress on my album cover classification project, highlighting lessons that would be useful to others who are getting started with deep learning.

                                                                                                                                                                                                        July 6, 2015

                                                                                                                                                                                                        Hopping on the deep learning bandwagon

                                                                                                                                                                                                        To become proficient at solving data science problems, you need to get your hands dirty. Here, I used album cover classification to learn about deep learning.

                                                                                                                                                                                                        June 6, 2015

                                                                                                                                                                                                        First steps in data science: author-aware sentiment analysis

                                                                                                                                                                                                        I became a data scientist by doing a PhD, but the same steps can be followed without a formal education program.

                                                                                                                                                                                                        May 2, 2015

                                                                                                                                                                                                        My divestment from fossil fuels

                                                                                                                                                                                                        Recent choices I’ve made to reduce my exposure to fossil fuels, including practical steps that can be taken by Australians and generally applicable lessons.

                                                                                                                                                                                                        April 24, 2015

                                                                                                                                                                                                        My PhD work

                                                                                                                                                                                                        An overview of my PhD in data science / artificial intelligence. Thesis title: Text Mining and Rating Prediction with Topical User Models.

                                                                                                                                                                                                        March 30, 2015

                                                                                                                                                                                                        The long road to a lifestyle business

                                                                                                                                                                                                        Progress since leaving my last full-time job and setting on an independent path that includes data science consulting and work on my own projects.

                                                                                                                                                                                                        March 22, 2015

                                                                                                                                                                                                        Learning to rank for personalised search (Yandex Search Personalisation – Kaggle Competition Summary – Part 2)

                                                                                                                                                                                                        My team’s solution to the Yandex Search Personalisation competition (finished 9th out of 194 teams).

                                                                                                                                                                                                        February 11, 2015

                                                                                                                                                                                                        Is thinking like a search engine possible? (Yandex search personalisation – Kaggle competition summary – part 1)

                                                                                                                                                                                                        Insights on search personalisation and SEO from participating in a Kaggle competition (finished 9th out of 194 teams).

                                                                                                                                                                                                        January 29, 2015

                                                                                                                                                                                                        Automating Parse.com bulk data imports

                                                                                                                                                                                                        A script for importing data into the Parse backend-as-a-service.

                                                                                                                                                                                                        January 15, 2015

                                                                                                                                                                                                        Stochastic Gradient Boosting: Choosing the Best Number of Iterations

                                                                                                                                                                                                        Exploring an approach to choosing the optimal number of iterations in stochastic gradient boosting, following a bug I found in scikit-learn.

                                                                                                                                                                                                        December 29, 2014

                                                                                                                                                                                                        SEO: Mostly about showing up?

                                                                                                                                                                                                        Increasing SEO traffic to BCRecommender by adding content and opening up more pages for crawling. It turns out that thin content is better than no content.

                                                                                                                                                                                                        December 15, 2014

                                                                                                                                                                                                        Fitting noise: Forecasting the sale price of bulldozers (Kaggle competition summary)

                                                                                                                                                                                                        Summary of a Kaggle competition to forecast bulldozer sale price, where I finished 9th out of 476 teams.

                                                                                                                                                                                                        November 19, 2014

                                                                                                                                                                                                        BCRecommender Traction Update

                                                                                                                                                                                                        Update on BCRecommender traction using three channels: blogger outreach, search engine optimisation, and content marketing.

                                                                                                                                                                                                        November 5, 2014

                                                                                                                                                                                                        What is data science?

                                                                                                                                                                                                        Data science has been a hot term in the past few years. Still, there isn’t a single definition of the field. This post discusses my favourite definition.

                                                                                                                                                                                                        October 23, 2014

                                                                                                                                                                                                        Greek Media Monitoring Kaggle competition: My approach

                                                                                                                                                                                                        Summary of my approach to the Greek Media Monitoring Kaggle competition, where I finished 6th out of 120 teams.

                                                                                                                                                                                                        October 7, 2014

                                                                                                                                                                                                        Applying the Traction Book’s Bullseye framework to BCRecommender

                                                                                                                                                                                                        Ranking 19 channels with the goal of getting traction for BCRecommender.

                                                                                                                                                                                                        September 24, 2014

                                                                                                                                                                                                        Bandcamp recommendation and discovery algorithms

                                                                                                                                                                                                        The recommendation backend for my BCRecommender service for personalised Bandcamp music discovery.

                                                                                                                                                                                                        September 19, 2014

                                                                                                                                                                                                        Building a recommender system on a shoestring budget (or: BCRecommender part 2 – general system layout)

                                                                                                                                                                                                        Iterating on my BCRecommender service with the goal of keeping costs low while providing a valuable music recommendation service.

                                                                                                                                                                                                        September 7, 2014

                                                                                                                                                                                                        Building a Bandcamp recommender system (part 1 – motivation)

                                                                                                                                                                                                        My motivation behind building BCRecommender, a free recommendation & discovery service for Bandcamp music.

                                                                                                                                                                                                        August 30, 2014

                                                                                                                                                                                                        How to (almost) win Kaggle competitions

                                                                                                                                                                                                        Summary of a talk I gave at the Data Science Sydney meetup with ten tips on almost-winning Kaggle competitions.

                                                                                                                                                                                                        August 24, 2014

                                                                                                                                                                                                        Data’s hierarchy of needs

                                                                                                                                                                                                        Discussing the hierarchy of needs proposed by Jay Kreps. Key takeaway: Data-driven algorithms & insights can only be as good as the underlying data.

                                                                                                                                                                                                        August 17, 2014

                                                                                                                                                                                                        Kaggle competition tips and summaries

                                                                                                                                                                                                        Pointers to all my Kaggle advice posts and competition summaries.

                                                                                                                                                                                                        April 5, 2014

                                                                                                                                                                                                        Kaggle beginner tips

                                                                                                                                                                                                        First post! An email I sent to members of the Data Science Sydney Meetup with tips on how to get started with Kaggle competitions.

                                                                                                                                                                                                        January 19, 2014
                                                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | +
                                                                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                        Is your tech stack ready for data-intensive applications?

                                                                                                                                                                                                        Questions to assess the quality of tech stacks and lifecycles, with a focus on artificial intelligence, machine learning, and analytics.

                                                                                                                                                                                                        June 24, 2024

                                                                                                                                                                                                        AI ain't gonna save you from bad data

                                                                                                                                                                                                        Since we’re far from a utopia where data issues are fully handled by AI, this post presents six questions humans can use to assess data projects.

                                                                                                                                                                                                        June 17, 2024

                                                                                                                                                                                                        Startup data health starts with healthy event tracking

                                                                                                                                                                                                        Expanding on the startup health check question of tracking Kukuyeva’s five business aspects as wide events.

                                                                                                                                                                                                        June 10, 2024

                                                                                                                                                                                                        How to avoid startups with poor development processes

                                                                                                                                                                                                        Questions that prospective data specialists and engineers should ask about development processes before accepting a startup role.

                                                                                                                                                                                                        June 3, 2024

                                                                                                                                                                                                        Plumbing, Decisions, and Automation: De-hyping Data & AI

                                                                                                                                                                                                        Three essential questions to understand where an organisation stands when it comes to Data & AI (with zero hype).

                                                                                                                                                                                                        May 27, 2024

                                                                                                                                                                                                        Question startup culture before accepting a data-to-AI role

                                                                                                                                                                                                        Eight questions that prospective data-to-AI employees should ask about a startup’s work and data culture.

                                                                                                                                                                                                        May 20, 2024

                                                                                                                                                                                                        Probing the People aspects of an early-stage startup

                                                                                                                                                                                                        Ten questions that prospective employees should ask about a startup’s team, especially for data-centric roles.

                                                                                                                                                                                                        May 13, 2024

                                                                                                                                                                                                        Business questions to ask before taking a startup data role

                                                                                                                                                                                                        Fourteen questions that prospective employees should ask about a startup’s business model and product, especially for data-focused roles.

                                                                                                                                                                                                        May 6, 2024

                                                                                                                                                                                                        Mentorship and the art of actionable advice

                                                                                                                                                                                                        Reflections on what it takes to package expertise and deliver timely, actionable advice outside the context of employee relationships.

                                                                                                                                                                                                        April 29, 2024

                                                                                                                                                                                                        Assessing a startup's data-to-AI health

                                                                                                                                                                                                        Reviewing the areas that should be assessed to determine a startup’s opportunities and challenges on the data/AI/ML front.

                                                                                                                                                                                                        April 22, 2024

                                                                                                                                                                                                        AI does not obviate the need for testing and observability

                                                                                                                                                                                                        It’s easy to prototype with AI, but production-grade AI apps require even more thorough testing and observability than traditional software.

                                                                                                                                                                                                        April 15, 2024

                                                                                                                                                                                                        My experience as a Data Tech Lead with Work on Climate

                                                                                                                                                                                                        The story of how I joined Work on Climate as a volunteer and became its data tech lead, with lessons applied to consulting & fractional work.

                                                                                                                                                                                                        April 8, 2024

                                                                                                                                                                                                        Artificial intelligence, automation, and the art of counting fish

                                                                                                                                                                                                        Discussing the use of AI to automate underwater marine surveys as an example of the uneven distribution of technological advancement.

                                                                                                                                                                                                        April 1, 2024

                                                                                                                                                                                                        Questions to consider when using AI for PDF data extraction

                                                                                                                                                                                                        Discussing considerations that arise when attempting to automate the extraction of structured data from PDFs and similar documents.

                                                                                                                                                                                                        March 11, 2024

                                                                                                                                                                                                        Two types of startup data problems

                                                                                                                                                                                                        Classifying startups as ML-centric or non-ML is a helpful exercise to uncover the data challenges they’re likely to face.

                                                                                                                                                                                                        March 4, 2024

                                                                                                                                                                                                        Avoiding AI complexity: First, write no code

                                                                                                                                                                                                        Two stories of getting AI functionality to production, which demonstrate the risks inherent in custom development versus starting with a no-code approach.

                                                                                                                                                                                                        February 26, 2024

                                                                                                                                                                                                        Building your startup's minimum viable data stack

                                                                                                                                                                                                        First post in a series on building a minimum viable data stack for startups, introducing key definitions, components, and considerations.

                                                                                                                                                                                                        February 19, 2024

                                                                                                                                                                                                        Nudging ChatGPT to invent books you have no time to read

                                                                                                                                                                                                        Getting ChatGPT Plus to elaborate on possible book content and produce a PDF cheatsheet, with the goal of learning about its capabilities.

                                                                                                                                                                                                        February 12, 2024

                                                                                                                                                                                                        Substance over titles: Your first data hire may be a data scientist

                                                                                                                                                                                                        Advice for hiring a startup’s first data person: match skills to business needs, consider contractors, and get help from data people.

                                                                                                                                                                                                        February 5, 2024

                                                                                                                                                                                                        New decade, new tagline: Data & AI for Impact

                                                                                                                                                                                                        Shifting focus to ‘Data & AI for Impact’, with more startup-related content, increased posting frequency, and deeper audience engagement.

                                                                                                                                                                                                        January 19, 2024

                                                                                                                                                                                                        Supporting volunteer monitoring of marine biodiversity with modern web and data tools

                                                                                                                                                                                                        Summarising the work Uri Seroussi and I did to improve Reef Life Survey’s Reef Species of the World app.

                                                                                                                                                                                                        November 29, 2023

                                                                                                                                                                                                        Lessons from reluctant data engineering

                                                                                                                                                                                                        Video and summary of a talk I gave at DataEngBytes Brisbane on what I learned from doing data engineering as part of every data science role I had.

                                                                                                                                                                                                        October 25, 2023

                                                                                                                                                                                                        My rediscovery of quiet writing on the open web

                                                                                                                                                                                                        Reflections on publishing on this website: Writing publicly to share thoughts and documentation beats chasing views and likes.

                                                                                                                                                                                                        August 28, 2023

                                                                                                                                                                                                        Was data science a failure mode of software engineering?

                                                                                                                                                                                                        Yes, data science projects have suffered from classic software engineering mistakes, but the field is maturing with the rise of new engineering roles.

                                                                                                                                                                                                        June 30, 2023

                                                                                                                                                                                                        How hackable are automated coding assessments?

                                                                                                                                                                                                        Exploring the hackability of speed-based coding tests, using CodeSignal’s Industry Coding Framework as a case study.

                                                                                                                                                                                                        May 26, 2023

                                                                                                                                                                                                        Remaining relevant as a small language model

                                                                                                                                                                                                        Bing Chat recently quipped that humans are small language models. Here are some of my thoughts on how we small language models can remain relevant (for now).

                                                                                                                                                                                                        April 21, 2023

                                                                                                                                                                                                        ChatGPT is transformative AI

                                                                                                                                                                                                        My perspective after a week of using ChatGPT: This is a step change in finding distilled information, and it’s only the beginning.

                                                                                                                                                                                                        December 11, 2022

                                                                                                                                                                                                        Causal Machine Learning is off to a good start, despite some issues

                                                                                                                                                                                                        Reviewing the first three chapters of the book Causal Machine Learning by Robert Osazuwa Ness.

                                                                                                                                                                                                        September 12, 2022

                                                                                                                                                                                                        The mission matters: Moving to climate tech as a data scientist

                                                                                                                                                                                                        Discussing my recent career move into climate tech as a way of doing more to help mitigate dangerous climate change.

                                                                                                                                                                                                        June 6, 2022

                                                                                                                                                                                                        Building useful machine learning tools keeps getting easier: A fish ID case study

                                                                                                                                                                                                        Lessons learned building a fish ID web app with fast.ai and Streamlit, in an attempt to reduce my fear of missing out on the latest deep learning developments.

                                                                                                                                                                                                        March 20, 2022

                                                                                                                                                                                                        Analysis strategies in online A/B experiments: Intention-to-treat, per-protocol, and other lessons from clinical trials

                                                                                                                                                                                                        Epidemiologists analyse clinical trials to estimate the intention-to-treat and per-protocol effects. This post applies their strategies to online experiments.

                                                                                                                                                                                                        January 14, 2022

                                                                                                                                                                                                        Use your human brain to avoid artificial intelligence disasters

                                                                                                                                                                                                        Overview of a talk I gave at a deep learning course, focusing on AI ethics as the need for humans to think on the context and consequences of applying AI.

                                                                                                                                                                                                        November 22, 2021

                                                                                                                                                                                                        Migrating from WordPress.com to Hugo on GitHub + Cloudflare

                                                                                                                                                                                                        My reasons for switching from WordPress.com to Hugo on GitHub + Cloudflare, along with a summary of the solution components and migration process.

                                                                                                                                                                                                        November 10, 2021

                                                                                                                                                                                                        My work with Automattic

                                                                                                                                                                                                        Back-dated meta-post that gathers my posts on Automattic blogs into a summary of the work I’ve done with the company.

                                                                                                                                                                                                        October 7, 2021

                                                                                                                                                                                                        Some highlights from 2020

                                                                                                                                                                                                        Sharing remote teamwork insights, my climate & sustainability activism, Reef Life Survey publications, and progress on Automattic’s Experimentation Platform.

                                                                                                                                                                                                        April 5, 2021

                                                                                                                                                                                                        Many is not enough: Counting simulations to bootstrap the right way

                                                                                                                                                                                                        Going deeper into correct testing of different methods for bootstrap estimation of confidence intervals.

                                                                                                                                                                                                        August 24, 2020

                                                                                                                                                                                                        Software commodities are eating interesting data science work

                                                                                                                                                                                                        Being a data scientist can sometimes feel like a race against software commodities that replace interesting work. What can one do to remain relevant?

                                                                                                                                                                                                        January 11, 2020

                                                                                                                                                                                                        A day in the life of a remote data scientist

                                                                                                                                                                                                        Video of a talk I gave on remote data science work at the Data Science Sydney meetup.

                                                                                                                                                                                                        December 11, 2019

                                                                                                                                                                                                        Bootstrapping the right way?

                                                                                                                                                                                                        Video and summary of a talk I gave at YOW! Data on bootstrap estimation of confidence intervals.

                                                                                                                                                                                                        October 6, 2019

                                                                                                                                                                                                        Hackers beware: Bootstrap sampling may be harmful

                                                                                                                                                                                                        Bootstrap sampling has been promoted as an easy way of modelling uncertainty to hackers without much statistical knowledge. But things aren’t that simple.

                                                                                                                                                                                                        January 7, 2019

                                                                                                                                                                                                        The most practical causal inference book I’ve read (is still a draft)

                                                                                                                                                                                                        Causal Inference by Miguel Hernán and Jamie Robins is a must-read for anyone interested in the area.

                                                                                                                                                                                                        December 24, 2018

                                                                                                                                                                                                        Reflections on remote data science work

                                                                                                                                                                                                        Discussing the pluses and minuses of remote work eighteen months after joining Automattic as a data scientist.

                                                                                                                                                                                                        November 3, 2018

                                                                                                                                                                                                        Defining data science in 2018

                                                                                                                                                                                                        Updating my definition of data science to match changes in the field. It is now broader than before, but its ultimate goal is still to support decisions.

                                                                                                                                                                                                        July 22, 2018

                                                                                                                                                                                                        Advice for aspiring data scientists and other FAQs

                                                                                                                                                                                                        Frequently asked questions by visitors to this site, especially around entering the data science field.

                                                                                                                                                                                                        October 15, 2017

                                                                                                                                                                                                        State of Bandcamp Recommender, Late 2017

                                                                                                                                                                                                        Call for BCRecommender maintainers followed by a decision to shut it down, as I don’t have enough time and Bandcamp now offers recommendations.

                                                                                                                                                                                                        September 2, 2017

                                                                                                                                                                                                        My 10-step path to becoming a remote data scientist with Automattic

                                                                                                                                                                                                        I wanted a well-paid data science-y remote job with an established company that offers a good life balance and makes products I care about. I got it eventually.

                                                                                                                                                                                                        July 29, 2017

                                                                                                                                                                                                        Exploring and visualising Reef Life Survey data

                                                                                                                                                                                                        Web tools I built to visualise Reef Life Survey data and assist citizen scientists in underwater visual census work.

                                                                                                                                                                                                        June 3, 2017

                                                                                                                                                                                                        Customer lifetime value and the proliferation of misinformation on the internet

                                                                                                                                                                                                        There’s a lot of misleading content on the estimation of customer lifetime value. Here’s what I learned about doing it well.

                                                                                                                                                                                                        January 8, 2017

                                                                                                                                                                                                        Ask Why! Finding motives, causes, and purpose in data science

                                                                                                                                                                                                        Video and summary of a talk I gave at the Data Science Sydney meetup, about going beyond the what & how of predictive modelling.

                                                                                                                                                                                                        September 19, 2016

                                                                                                                                                                                                        If you don’t pay attention, data can drive you off a cliff

                                                                                                                                                                                                        Seven common mistakes to avoid when working with data, such as ignoring uncertainty and confusing observed and unobserved quantities.

                                                                                                                                                                                                        August 21, 2016

                                                                                                                                                                                                        Is Data Scientist a useless job title?

                                                                                                                                                                                                        It seems like anyone who touches data can call themselves a data scientist, which makes the title useless. The work they do can still be useful, though.

                                                                                                                                                                                                        August 4, 2016

                                                                                                                                                                                                        Making Bayesian A/B testing more accessible

                                                                                                                                                                                                        A web tool I built to interpret A/B test results in a Bayesian way, including prior specification, visualisations, and decision rules.

                                                                                                                                                                                                        June 19, 2016

                                                                                                                                                                                                        Diving deeper into causality: Pearl, Kleinberg, Hill, and untested assumptions

                                                                                                                                                                                                        Discussing the need for untested assumptions and temporality in causal inference. Mostly based on Samantha Kleinberg’s Causality, Probability, and Time.

                                                                                                                                                                                                        May 14, 2016

                                                                                                                                                                                                        The rise of greedy robots

                                                                                                                                                                                                        Is artificial/machine intelligence a future threat? I argue that it’s already here, with greedy robots already dominating our lives.

                                                                                                                                                                                                        March 20, 2016

                                                                                                                                                                                                        Why you should stop worrying about deep learning and deepen your understanding of causality instead

                                                                                                                                                                                                        Causality is often overlooked but is of much higher relevance to most data scientists than deep learning.

                                                                                                                                                                                                        February 14, 2016

                                                                                                                                                                                                        The joys of offline data collection

                                                                                                                                                                                                        Insights on data collection and machine learning from spending a month sailing, diving, and counting fish with Reef Life Survey.

                                                                                                                                                                                                        January 24, 2016

                                                                                                                                                                                                        This holiday season, give me real insights

                                                                                                                                                                                                        Some companies present raw data or information as “insights”. This post surveys some examples, and discusses how they can be turned into real insights.

                                                                                                                                                                                                        December 8, 2015

                                                                                                                                                                                                        The hardest parts of data science

                                                                                                                                                                                                        Defining feasible problems and coming up with reasonable ways of measuring solutions is harder than building accurate models or obtaining clean data.

                                                                                                                                                                                                        November 23, 2015

                                                                                                                                                                                                        Migrating a simple web application from MongoDB to Elasticsearch

                                                                                                                                                                                                        Migrating BCRecommender from MongoDB to Elasticsearch made it possible to offer a richer search experience to users at a similar cost, among other benefits.

                                                                                                                                                                                                        November 4, 2015

                                                                                                                                                                                                        Miscommunicating science: Simplistic models, nutritionism, and the art of storytelling

                                                                                                                                                                                                        Nutritionism is a special case of misinterpretation and miscommunication of scientific results – something many data scientists encounter in their work.

                                                                                                                                                                                                        October 19, 2015

                                                                                                                                                                                                        The wonderful world of recommender systems

                                                                                                                                                                                                        Giving an overview of the field and common paradigms, and debunking five common myths about recommender systems.

                                                                                                                                                                                                        October 2, 2015

                                                                                                                                                                                                        You don’t need a data scientist (yet)

                                                                                                                                                                                                        Hiring data scientists prematurely is wasteful and frustrating. Here are some questions to ask before you hire your first data scientist.

                                                                                                                                                                                                        August 24, 2015

                                                                                                                                                                                                        Goodbye, Parse.com

                                                                                                                                                                                                        Migrating my web apps away from Parse.com due to reliability issues. Self-hosting is a better solution.

                                                                                                                                                                                                        July 31, 2015

                                                                                                                                                                                                        Learning about deep learning through album cover classification

                                                                                                                                                                                                        Progress on my album cover classification project, highlighting lessons that would be useful to others who are getting started with deep learning.

                                                                                                                                                                                                        July 6, 2015

                                                                                                                                                                                                        Hopping on the deep learning bandwagon

                                                                                                                                                                                                        To become proficient at solving data science problems, you need to get your hands dirty. Here, I used album cover classification to learn about deep learning.

                                                                                                                                                                                                        June 6, 2015

                                                                                                                                                                                                        First steps in data science: author-aware sentiment analysis

                                                                                                                                                                                                        I became a data scientist by doing a PhD, but the same steps can be followed without a formal education program.

                                                                                                                                                                                                        May 2, 2015

                                                                                                                                                                                                        My divestment from fossil fuels

                                                                                                                                                                                                        Recent choices I’ve made to reduce my exposure to fossil fuels, including practical steps that can be taken by Australians and generally applicable lessons.

                                                                                                                                                                                                        April 24, 2015

                                                                                                                                                                                                        My PhD work

                                                                                                                                                                                                        An overview of my PhD in data science / artificial intelligence. Thesis title: Text Mining and Rating Prediction with Topical User Models.

                                                                                                                                                                                                        March 30, 2015

                                                                                                                                                                                                        The long road to a lifestyle business

                                                                                                                                                                                                        Progress since leaving my last full-time job and setting on an independent path that includes data science consulting and work on my own projects.

                                                                                                                                                                                                        March 22, 2015

                                                                                                                                                                                                        Learning to rank for personalised search (Yandex Search Personalisation – Kaggle Competition Summary – Part 2)

                                                                                                                                                                                                        My team’s solution to the Yandex Search Personalisation competition (finished 9th out of 194 teams).

                                                                                                                                                                                                        February 11, 2015

                                                                                                                                                                                                        Is thinking like a search engine possible? (Yandex search personalisation – Kaggle competition summary – part 1)

                                                                                                                                                                                                        Insights on search personalisation and SEO from participating in a Kaggle competition (finished 9th out of 194 teams).

                                                                                                                                                                                                        January 29, 2015

                                                                                                                                                                                                        Automating Parse.com bulk data imports

                                                                                                                                                                                                        A script for importing data into the Parse backend-as-a-service.

                                                                                                                                                                                                        January 15, 2015

                                                                                                                                                                                                        Stochastic Gradient Boosting: Choosing the Best Number of Iterations

                                                                                                                                                                                                        Exploring an approach to choosing the optimal number of iterations in stochastic gradient boosting, following a bug I found in scikit-learn.

                                                                                                                                                                                                        December 29, 2014

                                                                                                                                                                                                        SEO: Mostly about showing up?

                                                                                                                                                                                                        Increasing SEO traffic to BCRecommender by adding content and opening up more pages for crawling. It turns out that thin content is better than no content.

                                                                                                                                                                                                        December 15, 2014

                                                                                                                                                                                                        Fitting noise: Forecasting the sale price of bulldozers (Kaggle competition summary)

                                                                                                                                                                                                        Summary of a Kaggle competition to forecast bulldozer sale price, where I finished 9th out of 476 teams.

                                                                                                                                                                                                        November 19, 2014

                                                                                                                                                                                                        BCRecommender Traction Update

                                                                                                                                                                                                        Update on BCRecommender traction using three channels: blogger outreach, search engine optimisation, and content marketing.

                                                                                                                                                                                                        November 5, 2014

                                                                                                                                                                                                        What is data science?

                                                                                                                                                                                                        Data science has been a hot term in the past few years. Still, there isn’t a single definition of the field. This post discusses my favourite definition.

                                                                                                                                                                                                        October 23, 2014

                                                                                                                                                                                                        Greek Media Monitoring Kaggle competition: My approach

                                                                                                                                                                                                        Summary of my approach to the Greek Media Monitoring Kaggle competition, where I finished 6th out of 120 teams.

                                                                                                                                                                                                        October 7, 2014

                                                                                                                                                                                                        Applying the Traction Book’s Bullseye framework to BCRecommender

                                                                                                                                                                                                        Ranking 19 channels with the goal of getting traction for BCRecommender.

                                                                                                                                                                                                        September 24, 2014

                                                                                                                                                                                                        Bandcamp recommendation and discovery algorithms

                                                                                                                                                                                                        The recommendation backend for my BCRecommender service for personalised Bandcamp music discovery.

                                                                                                                                                                                                        September 19, 2014

                                                                                                                                                                                                        Building a recommender system on a shoestring budget (or: BCRecommender part 2 – general system layout)

                                                                                                                                                                                                        Iterating on my BCRecommender service with the goal of keeping costs low while providing a valuable music recommendation service.

                                                                                                                                                                                                        September 7, 2014

                                                                                                                                                                                                        Building a Bandcamp recommender system (part 1 – motivation)

                                                                                                                                                                                                        My motivation behind building BCRecommender, a free recommendation & discovery service for Bandcamp music.

                                                                                                                                                                                                        August 30, 2014

                                                                                                                                                                                                        How to (almost) win Kaggle competitions

                                                                                                                                                                                                        Summary of a talk I gave at the Data Science Sydney meetup with ten tips on almost-winning Kaggle competitions.

                                                                                                                                                                                                        August 24, 2014

                                                                                                                                                                                                        Data’s hierarchy of needs

                                                                                                                                                                                                        Discussing the hierarchy of needs proposed by Jay Kreps. Key takeaway: Data-driven algorithms & insights can only be as good as the underlying data.

                                                                                                                                                                                                        August 17, 2014

                                                                                                                                                                                                        Kaggle competition tips and summaries

                                                                                                                                                                                                        Pointers to all my Kaggle advice posts and competition summaries.

                                                                                                                                                                                                        April 5, 2014

                                                                                                                                                                                                        Kaggle beginner tips

                                                                                                                                                                                                        First post! An email I sent to members of the Data Science Sydney Meetup with tips on how to get started with Kaggle competitions.

                                                                                                                                                                                                        January 19, 2014
                                                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & PaperMod
                                                                                                                                                                                                        \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index b7295f26e..0ff2eed0d 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -1 +1 @@ -https://yanirseroussi.com/tags/business/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/career/2024-06-26T10:45:15+10:00https://yanirseroussi.com/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/quotes/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/startups/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/2024-06-26T10:45:15+10:00https://yanirseroussi.com/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/analytics/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/artificial-intelligence/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/data-science/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/data-strategy/2024-06-24T14:12:50+10:00https://yanirseroussi.com/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/machine-learning/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/software-engineering/2024-06-24T14:12:50+10:00https://yanirseroussi.com/til/2024/06/22/dealing-with-endless-data-changes/2024-06-23T08:52:50+10:00https://yanirseroussi.com/tags/devops/2024-06-23T08:52:50+10:00https://yanirseroussi.com/2024/06/17/ai-aint-gonna-save-you-from-bad-data/2024-06-17T13:13:44+10:00https://yanirseroussi.com/tags/books/2024-06-12T12:58:06+10:00https://yanirseroussi.com/til/2024/06/12/the-rules-of-the-passion-economy/2024-06-12T12:58:06+10:00https://yanirseroussi.com/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/2024-06-10T14:23:12+10:00https://yanirseroussi.com/2024/06/03/how-to-avoid-startups-with-poor-development-processes/2024-06-03T12:58:00+10:00https://yanirseroussi.com/tags/data-engineering/2024-05-27T12:25:30+10:00https://yanirseroussi.com/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/2024-05-27T12:25:30+10:00https://yanirseroussi.com/til/2024/05/25/adapting-to-the-economy-of-algorithms/2024-05-25T10:00:56+10:00https://yanirseroussi.com/tags/futurism/2024-05-25T10:00:56+10:00https://yanirseroussi.com/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/2024-05-21T17:08:32+10:00https://yanirseroussi.com/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/2024-05-13T12:41:01+10:00https://yanirseroussi.com/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/2024-05-06T14:41:43+10:00https://yanirseroussi.com/tags/consulting/2024-04-29T17:25:28+10:00https://yanirseroussi.com/2024/04/29/mentorship-and-the-art-of-actionable-advice/2024-04-29T17:25:28+10:00https://yanirseroussi.com/tags/personal/2024-04-29T17:25:28+10:00https://yanirseroussi.com/2024/04/22/assessing-a-startups-data-to-ai-health/2024-04-22T17:38:21+10:00https://yanirseroussi.com/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/2024-04-15T15:54:17+10:00https://yanirseroussi.com/tags/linkedin/2024-04-11T13:42:58+10:00https://yanirseroussi.com/til/2024/04/11/linkedin-is-a-teachable-skill/2024-04-11T13:42:58+10:00https://yanirseroussi.com/tags/marketing/2024-04-11T13:42:58+10:00https://yanirseroussi.com/tags/climate-change/2024-04-08T12:13:47+10:00https://yanirseroussi.com/tags/environment/2024-04-08T12:13:47+10:00https://yanirseroussi.com/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/2024-04-08T12:13:47+10:00https://yanirseroussi.com/tags/remote-work/2024-04-08T12:13:47+10:00https://yanirseroussi.com/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/2024-04-05T11:23:38+10:00https://yanirseroussi.com/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/2024-04-01T17:02:44+10:00https://yanirseroussi.com/tags/marine-science/2024-04-01T17:02:44+10:00https://yanirseroussi.com/tags/reef-life-survey/2024-04-01T17:02:44+10:00https://yanirseroussi.com/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/2024-03-12T16:33:48+10:00https://yanirseroussi.com/tags/productivity/2024-03-12T16:33:48+10:00https://yanirseroussi.com/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/2024-03-11T15:53:13+10:00https://yanirseroussi.com/2024/03/04/two-types-of-startup-data-problems/2024-03-05T08:47:19+10:00https://yanirseroussi.com/2024/02/26/avoiding-ai-complexity-first-write-no-code/2024-03-04T12:39:10+10:00https://yanirseroussi.com/2024/02/19/building-your-startups-minimum-viable-data-stack/2024-02-19T11:25:54+10:00https://yanirseroussi.com/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/2024-02-17T12:34:00+10:00https://yanirseroussi.com/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/2024-02-13T08:24:54+10:00https://yanirseroussi.com/til/2024/02/06/future-software-development-may-require-fewer-humans/2024-02-06T16:39:35+10:00https://yanirseroussi.com/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/2024-02-19T11:25:54+10:00https://yanirseroussi.com/tags/blogging/2024-01-19T16:35:09+10:00https://yanirseroussi.com/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/2024-01-19T16:35:09+10:00https://yanirseroussi.com/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/2024-01-09T13:23:28+10:00https://yanirseroussi.com/til/2024/01/08/the-power-of-parasocial-relationships/2024-01-08T16:31:22+10:00https://yanirseroussi.com/tags/data-business/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/2023-12-18T10:38:56+10:00https://yanirseroussi.com/tags/energy-markets/2023-12-14T10:46:41+10:00https://yanirseroussi.com/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/2023-12-14T10:46:41+10:00https://yanirseroussi.com/tags/data-visualisation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/web-development/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/2023-11-21T16:12:27+10:00https://yanirseroussi.com/2023/10/25/lessons-from-reluctant-data-engineering/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/2023-10-06T15:11:27+10:00https://yanirseroussi.com/tags/ethics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/2023-09-25T11:15:26+10:00https://yanirseroussi.com/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/2023-09-22T07:54:13+10:00https://yanirseroussi.com/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/14/email-notifications-on-public-github-commits/2023-08-14T15:44:21+10:00https://yanirseroussi.com/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/2023-08-11T14:35:20+10:00https://yanirseroussi.com/tags/github/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/security/2023-07-25T09:30:43+10:00https://yanirseroussi.com/til/2023/07/23/using-yubikey-for-ssh-access/2023-07-25T09:30:43+10:00https://yanirseroussi.com/tags/hugo/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/2023-07-17T17:18:06+10:00https://yanirseroussi.com/til/2023/07/11/you-cant-save-time/2024-03-12T16:33:31+10:00https://yanirseroussi.com/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/hackers/2024-06-19T17:03:21+10:00https://yanirseroussi.com/2023/05/26/how-hackable-are-automated-coding-assessments/2024-06-19T17:03:21+10:00https://yanirseroussi.com/tags/machine-intelligence/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a-small-language-model/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/12/11/chatgpt-is-transformative-ai/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/causal-inference/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2022/09/12/causal-machine-learning-book-draft-review/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/orkestra/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/politics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/sustainability/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/deep-learning/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/fast.ai/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/01/14/analysis-strategies-in-online-a-b-experiments/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/split-testing/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/statistics/2024-05-06T16:35:22+10:00https://yanirseroussi.com/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/cloudflare/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/wordpress/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/10/07/my-work-with-automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/04/05/some-highlights-from-2020/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/bootstrapping/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/confidence-intervals/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2020/01/11/software-commodities-are-eating-interesting-data-science-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2019/10/06/bootstrapping-the-right-way/2024-05-06T16:35:22+10:00https://yanirseroussi.com/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/11/03/reflections-on-remote-data-science-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/07/22/defining-data-science-in-2018/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/frequently-asked-questions/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/bandcamp/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/bcrecommender/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/09/02/state-of-bandcamp-recommender/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/elasticsearch/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/06/03/exploring-and-visualising-reef-life-survey-data/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/javascript/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/predictive-modelling/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/science-communication/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/search-engine-optimisation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/insights/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/08/04/is-data-scientist-a-useless-job-title/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/06/19/making-bayesian-ab-testing-more-accessible/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/economics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/03/20/the-rise-of-greedy-robots/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/scuba-diving/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/01/24/the-joys-of-offline-data-collection/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/facebook/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/12/08/this-holiday-season-give-me-real-insights/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/kaggle/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/11/23/the-hardest-parts-of-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/mongodb/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/health/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/nutrition/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/nutritionism/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/recommender-systems/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/10/02/the-wonderful-world-of-recommender-systems/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/08/24/you-dont-need-a-data-scientist-yet/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/07/31/goodbye-parse-com/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/parse.com/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/07/06/learning-about-deep-learning-through-album-cover-classification/2024-01-16T09:56:03+10:00https://yanirseroussi.com/deep-learning-resources/2021-11-09T15:38:25+10:00https://yanirseroussi.com/2015/06/06/hopping-on-the-deep-learning-bandwagon/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/sentiment-analysis/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/divestment/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/fossil-fuels/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/04/24/my-divestment-from-fossil-fuels/2024-01-16T09:56:03+10:00https://yanirseroussi.com/phd-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/03/22/the-long-road-to-a-lifestyle-business/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/gradient-boosting/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/kaggle-competition/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/01/15/automating-parse-com-bulk-data-imports/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/phantomjs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/scikit-learn/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/12/15/seo-mostly-about-showing-up/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/traction-book/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/price-forecasting/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/11/05/bcrecommender-traction-update/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/music/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/10/23/what-is-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/multi-label-classification/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/music-industry/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/08/24/how-to-almost-win-kaggle-competitions/2023-07-06T09:28:02+10:00https://yanirseroussi.com/tags/kaggle-beginners/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/08/17/datas-hierarchy-of-needs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/kaggle/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/01/19/kaggle-beginner-tips/2023-07-06T09:28:02+10:00https://yanirseroussi.com/about/2024-03-08T11:21:16+10:00https://yanirseroussi.com/free-intro-call/2024-05-22T17:54:36+10:00https://yanirseroussi.com/posts/2024-05-09T10:03:31+10:00https://yanirseroussi.com/causal-inference-resources/2023-07-06T16:01:57+10:00https://yanirseroussi.com/consult/2024-05-23T15:31:11+10:00https://yanirseroussi.com/data-to-ai-health-check/2024-05-22T17:53:56+10:00https://yanirseroussi.com/contact/2024-05-23T15:31:11+10:00https://yanirseroussi.com/talks/2024-05-06T16:35:22+10:00https://yanirseroussi.com/til/2024-05-09T10:03:31+10:00 \ No newline at end of file +https://yanirseroussi.com/tags/business/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/career/2024-06-26T10:45:15+10:00https://yanirseroussi.com/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/quotes/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/startups/2024-06-26T10:45:15+10:00https://yanirseroussi.com/tags/2024-06-26T10:45:15+10:00https://yanirseroussi.com/2024-06-26T12:57:51+10:00https://yanirseroussi.com/tags/analytics/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/artificial-intelligence/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/data-science/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/data-strategy/2024-06-24T14:12:50+10:00https://yanirseroussi.com/2024/06/24/is-your-tech-stack-ready-for-data-intensive-applications/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/machine-learning/2024-06-24T14:12:50+10:00https://yanirseroussi.com/tags/software-engineering/2024-06-24T14:12:50+10:00https://yanirseroussi.com/til/2024/06/22/dealing-with-endless-data-changes/2024-06-23T08:52:50+10:00https://yanirseroussi.com/tags/devops/2024-06-23T08:52:50+10:00https://yanirseroussi.com/2024/06/17/ai-aint-gonna-save-you-from-bad-data/2024-06-17T13:13:44+10:00https://yanirseroussi.com/tags/books/2024-06-12T12:58:06+10:00https://yanirseroussi.com/til/2024/06/12/the-rules-of-the-passion-economy/2024-06-12T12:58:06+10:00https://yanirseroussi.com/2024/06/10/startup-data-health-starts-with-healthy-event-tracking/2024-06-10T14:23:12+10:00https://yanirseroussi.com/2024/06/03/how-to-avoid-startups-with-poor-development-processes/2024-06-03T12:58:00+10:00https://yanirseroussi.com/tags/data-engineering/2024-05-27T12:25:30+10:00https://yanirseroussi.com/2024/05/27/plumbing-decisions-and-automation-de-hyping-data-and-ai/2024-05-27T12:25:30+10:00https://yanirseroussi.com/til/2024/05/25/adapting-to-the-economy-of-algorithms/2024-05-25T10:00:56+10:00https://yanirseroussi.com/tags/futurism/2024-05-25T10:00:56+10:00https://yanirseroussi.com/2024/05/20/question-startup-culture-before-accepting-a-data-to-ai-role/2024-05-21T17:08:32+10:00https://yanirseroussi.com/2024/05/13/probing-the-people-aspects-of-an-early-stage-startup/2024-05-13T12:41:01+10:00https://yanirseroussi.com/2024/05/06/business-questions-to-ask-before-taking-a-startup-data-role/2024-05-06T14:41:43+10:00https://yanirseroussi.com/tags/consulting/2024-04-29T17:25:28+10:00https://yanirseroussi.com/2024/04/29/mentorship-and-the-art-of-actionable-advice/2024-04-29T17:25:28+10:00https://yanirseroussi.com/tags/personal/2024-04-29T17:25:28+10:00https://yanirseroussi.com/2024/04/22/assessing-a-startups-data-to-ai-health/2024-04-22T17:38:21+10:00https://yanirseroussi.com/2024/04/15/ai-does-not-obviate-the-need-for-testing-and-observability/2024-04-15T15:54:17+10:00https://yanirseroussi.com/tags/linkedin/2024-04-11T13:42:58+10:00https://yanirseroussi.com/til/2024/04/11/linkedin-is-a-teachable-skill/2024-04-11T13:42:58+10:00https://yanirseroussi.com/tags/marketing/2024-04-11T13:42:58+10:00https://yanirseroussi.com/tags/climate-change/2024-04-08T12:13:47+10:00https://yanirseroussi.com/tags/environment/2024-04-08T12:13:47+10:00https://yanirseroussi.com/2024/04/08/my-experience-as-a-data-tech-lead-with-work-on-climate/2024-04-08T12:13:47+10:00https://yanirseroussi.com/tags/remote-work/2024-04-08T12:13:47+10:00https://yanirseroussi.com/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/2024-04-05T11:23:38+10:00https://yanirseroussi.com/2024/04/01/artificial-intelligence-automation-and-the-art-of-counting-fish/2024-04-01T17:02:44+10:00https://yanirseroussi.com/tags/marine-science/2024-04-01T17:02:44+10:00https://yanirseroussi.com/tags/reef-life-survey/2024-04-01T17:02:44+10:00https://yanirseroussi.com/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/2024-03-12T16:33:48+10:00https://yanirseroussi.com/tags/productivity/2024-03-12T16:33:48+10:00https://yanirseroussi.com/2024/03/11/questions-to-consider-when-using-ai-for-pdf-data-extraction/2024-03-11T15:53:13+10:00https://yanirseroussi.com/2024/03/04/two-types-of-startup-data-problems/2024-03-05T08:47:19+10:00https://yanirseroussi.com/2024/02/26/avoiding-ai-complexity-first-write-no-code/2024-03-04T12:39:10+10:00https://yanirseroussi.com/2024/02/19/building-your-startups-minimum-viable-data-stack/2024-02-19T11:25:54+10:00https://yanirseroussi.com/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/2024-02-17T12:34:00+10:00https://yanirseroussi.com/2024/02/12/nudging-chatgpt-to-invent-books-you-have-no-time-to-read/2024-02-13T08:24:54+10:00https://yanirseroussi.com/til/2024/02/06/future-software-development-may-require-fewer-humans/2024-02-06T16:39:35+10:00https://yanirseroussi.com/2024/02/05/substance-over-titles-your-first-data-hire-may-be-a-data-scientist/2024-02-19T11:25:54+10:00https://yanirseroussi.com/tags/blogging/2024-01-19T16:35:09+10:00https://yanirseroussi.com/2024/01/19/new-decade-new-tagline-data-and-ai-for-impact/2024-01-19T16:35:09+10:00https://yanirseroussi.com/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/2024-01-09T13:23:28+10:00https://yanirseroussi.com/til/2024/01/08/the-power-of-parasocial-relationships/2024-01-08T16:31:22+10:00https://yanirseroussi.com/tags/data-business/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/2023-12-18T10:38:56+10:00https://yanirseroussi.com/tags/energy-markets/2023-12-14T10:46:41+10:00https://yanirseroussi.com/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/2023-12-14T10:46:41+10:00https://yanirseroussi.com/tags/data-visualisation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2023/11/29/supporting-volunteer-monitoring-of-marine-biodiversity-with-modern-web-and-data-tools/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/web-development/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/2023-11-21T16:12:27+10:00https://yanirseroussi.com/2023/10/25/lessons-from-reluctant-data-engineering/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/2023-10-06T15:11:27+10:00https://yanirseroussi.com/tags/ethics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/2023-09-25T11:15:26+10:00https://yanirseroussi.com/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/2023-09-22T07:54:13+10:00https://yanirseroussi.com/2023/08/28/my-rediscovery-of-quiet-writing-on-the-open-web/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/2024-03-12T16:33:31+10:00https://yanirseroussi.com/til/2023/08/14/email-notifications-on-public-github-commits/2023-08-14T15:44:21+10:00https://yanirseroussi.com/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/2023-08-11T14:35:20+10:00https://yanirseroussi.com/tags/github/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/security/2023-07-25T09:30:43+10:00https://yanirseroussi.com/til/2023/07/23/using-yubikey-for-ssh-access/2023-07-25T09:30:43+10:00https://yanirseroussi.com/tags/hugo/2024-01-16T09:56:03+10:00https://yanirseroussi.com/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/2023-07-17T17:18:06+10:00https://yanirseroussi.com/til/2023/07/11/you-cant-save-time/2024-03-12T16:33:31+10:00https://yanirseroussi.com/2023/06/30/was-data-science-a-failure-mode-of-software-engineering/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/hackers/2024-06-19T17:03:21+10:00https://yanirseroussi.com/2023/05/26/how-hackable-are-automated-coding-assessments/2024-06-19T17:03:21+10:00https://yanirseroussi.com/tags/machine-intelligence/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2023/04/21/remaining-relevant-as-a-small-language-model/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/12/11/chatgpt-is-transformative-ai/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/causal-inference/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2022/09/12/causal-machine-learning-book-draft-review/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/orkestra/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/politics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/sustainability/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2022/06/06/the-mission-matters-moving-to-climate-tech-as-a-data-scientist/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/03/20/building-useful-machine-learning-tools-keeps-getting-easier-a-fish-id-case-study/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/deep-learning/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/fast.ai/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2022/01/14/analysis-strategies-in-online-a-b-experiments/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/split-testing/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/statistics/2024-05-06T16:35:22+10:00https://yanirseroussi.com/2021/11/22/use-your-human-brain-to-avoid-artificial-intelligence-disasters/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/cloudflare/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/11/10/migrating-from-wordpress-com-to-hugo-on-github-cloudflare/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/wordpress/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/10/07/my-work-with-automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2021/04/05/some-highlights-from-2020/2024-02-21T11:52:55+10:00https://yanirseroussi.com/tags/bootstrapping/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/confidence-intervals/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2020/08/24/many-is-not-enough-counting-simulations-to-bootstrap-the-right-way/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2020/01/11/software-commodities-are-eating-interesting-data-science-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2019/12/12/a-day-in-the-life-of-a-remote-data-scientist/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2019/10/06/bootstrapping-the-right-way/2024-05-06T16:35:22+10:00https://yanirseroussi.com/2019/01/08/hackers-beware-bootstrap-sampling-may-be-harmful/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/12/24/the-most-practical-causal-inference-book-ive-read-is-still-a-draft/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/11/03/reflections-on-remote-data-science-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2018/07/22/defining-data-science-in-2018/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/10/15/advice-for-aspiring-data-scientists-and-other-faqs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/frequently-asked-questions/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/bandcamp/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/bcrecommender/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/09/02/state-of-bandcamp-recommender/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/elasticsearch/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/07/29/my-10-step-path-to-becoming-a-remote-data-scientist-with-automattic/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/06/03/exploring-and-visualising-reef-life-survey-data/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/javascript/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2017/01/08/customer-lifetime-value-and-the-proliferation-of-misinformation-on-the-internet/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/predictive-modelling/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/science-communication/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/search-engine-optimisation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/09/19/ask-why-finding-motives-causes-and-purpose-in-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/insights/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/08/21/seven-ways-to-be-data-driven-off-a-cliff/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/08/04/is-data-scientist-a-useless-job-title/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/06/19/making-bayesian-ab-testing-more-accessible/2024-02-21T11:52:55+10:00https://yanirseroussi.com/2016/05/15/diving-deeper-into-causality-pearl-kleinberg-hill-and-untested-assumptions/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/economics/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/03/20/the-rise-of-greedy-robots/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/02/14/why-you-should-stop-worrying-about-deep-learning-and-deepen-your-understanding-of-causality-instead/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/scuba-diving/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2016/01/24/the-joys-of-offline-data-collection/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/facebook/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/12/08/this-holiday-season-give-me-real-insights/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/kaggle/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/11/23/the-hardest-parts-of-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/11/04/migrating-a-simple-web-application-from-mongodb-to-elasticsearch/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/mongodb/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/health/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/10/19/nutritionism-and-the-need-for-complex-models-to-explain-complex-phenomena/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/nutrition/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/nutritionism/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/recommender-systems/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/10/02/the-wonderful-world-of-recommender-systems/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/08/24/you-dont-need-a-data-scientist-yet/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/07/31/goodbye-parse-com/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/parse.com/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/07/06/learning-about-deep-learning-through-album-cover-classification/2024-01-16T09:56:03+10:00https://yanirseroussi.com/deep-learning-resources/2021-11-09T15:38:25+10:00https://yanirseroussi.com/2015/06/06/hopping-on-the-deep-learning-bandwagon/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/05/02/first-steps-in-data-science-author-aware-sentiment-analysis/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/sentiment-analysis/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/divestment/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/fossil-fuels/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/04/24/my-divestment-from-fossil-fuels/2024-01-16T09:56:03+10:00https://yanirseroussi.com/phd-work/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/03/22/the-long-road-to-a-lifestyle-business/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/gradient-boosting/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/kaggle-competition/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/02/11/learning-to-rank-for-personalised-search-yandex-search-personalisation-kaggle-competition-summary-part-2/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/01/29/is-thinking-like-a-search-engine-possible-yandex-search-personalisation-kaggle-competition-summary-part-1/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2015/01/15/automating-parse-com-bulk-data-imports/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/phantomjs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/scikit-learn/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/12/15/seo-mostly-about-showing-up/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/traction-book/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/11/19/fitting-noise-forecasting-the-sale-price-of-bulldozers-kaggle-competition-summary/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/price-forecasting/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/11/05/bcrecommender-traction-update/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/music/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/10/23/what-is-data-science/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/10/07/greek-media-monitoring-kaggle-competition-my-approach/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/multi-label-classification/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/09/24/applying-the-traction-books-bullseye-framework-to-bcrecommender/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/09/19/bandcamp-recommendation-and-discovery-algorithms/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/09/07/building-a-recommender-system-on-a-shoestring-budget/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/08/30/building-a-bandcamp-recommender-system-part-1-motivation/2024-01-16T09:56:03+10:00https://yanirseroussi.com/tags/music-industry/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/08/24/how-to-almost-win-kaggle-competitions/2023-07-06T09:28:02+10:00https://yanirseroussi.com/tags/kaggle-beginners/2023-07-06T09:28:02+10:00https://yanirseroussi.com/2014/08/17/datas-hierarchy-of-needs/2024-01-16T09:56:03+10:00https://yanirseroussi.com/kaggle/2024-01-16T09:56:03+10:00https://yanirseroussi.com/2014/01/19/kaggle-beginner-tips/2023-07-06T09:28:02+10:00https://yanirseroussi.com/about/2024-03-08T11:21:16+10:00https://yanirseroussi.com/free-intro-call/2024-06-26T12:57:51+10:00https://yanirseroussi.com/posts/2024-05-09T10:03:31+10:00https://yanirseroussi.com/causal-inference-resources/2023-07-06T16:01:57+10:00https://yanirseroussi.com/consult/2024-05-23T15:31:11+10:00https://yanirseroussi.com/data-to-ai-health-check/2024-06-26T12:57:51+10:00https://yanirseroussi.com/contact/2024-05-23T15:31:11+10:00https://yanirseroussi.com/talks/2024-05-06T16:35:22+10:00https://yanirseroussi.com/til/2024-05-09T10:03:31+10:00 \ No newline at end of file diff --git a/til/2023/07/11/you-cant-save-time/index.html b/til/2023/07/11/you-cant-save-time/index.html index de2a12625..b7c892420 100644 --- a/til/2023/07/11/you-cant-save-time/index.html +++ b/til/2023/07/11/you-cant-save-time/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                        Home » TIL: Today I learned...

                                                                                                                                                                                                        You can't save time

                                                                                                                                                                                                        July 11, 2023

                                                                                                                                                                                                        Quoting How to Speak Whale: A Voyage into the Future of Animal Communication by Tom Mustill (wonderful book!):

                                                                                                                                                                                                        I was gobsmacked. A whale lands on you and disappears. End of story. But thanks to lots of people who liked looking at whales and their intelligent machines, it was not the end at all. Machine learning and other branches of AI influence our daily lives in myriad ways. They have helped this book come into existence, with an algorithm transcribing the hundreds of hours of interviews I conducted for it. Other algorithms have checked my spelling and finished my sentences for me as I typed them. Google’s effective prediction of my email responses has made me realize how predictable a lot of my writing is (sorry, reader), and by extension, perhaps, most human language is. It has saved me huge amounts of time, and I have ended up spending this saved time procrastinating by looking at my phone, at news apps and shopping sites and social media, all of which have been beautifully designed and pumped full of AI whose purpose is to suck up my time and money and data.

                                                                                                                                                                                                        Related:

                                                                                                                                                                                                        Subscribe
                                                                                                                                                                                                          -
                                                                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/index.html b/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/index.html index a1fad6733..d37c66ef7 100644 --- a/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/index.html +++ b/til/2023/07/17/making-a-til-section-with-hugo-and-papermod/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                          Home » TIL: Today I learned...

                                                                                                                                                                                                          Making a TIL section with Hugo and PaperMod

                                                                                                                                                                                                          July 17, 2023

                                                                                                                                                                                                          I started following Simon Willison as a way of getting informed about practical applications of large language models. It turns out that Simon is also an incredibly prolific blogger and open source contributor, with posts dating back to 2002. He recently gave an interview titled The Data Enthusiast’s Toolkit, where he discussed his Datasette project and a bit about his approach to writing online. In addition to his main blog, he maintains a separate TIL subdomain, where shares things he’s learned in quick rough posts.

                                                                                                                                                                                                          Most of my main website posts take a while to write, so I post infrequently. I enjoy the process of learning about topics through public writing, as I often start out thinking things are a certain way, and then make new discoveries when I search for references. However, I liked the idea of sharing more through quicker posts, so I decided to add a TIL section to my site.

                                                                                                                                                                                                          I made the switch from WordPress.com to Hugo almost two years ago and haven’t looked back. I love the extra control that it gives me, though it sometimes requires a bit of tinkering. However, adding a TIL section was a breeze thanks to a post by Jacob Kaplan-Moss. Since I use the PaperMod theme and my site is set up differently, I followed slightly different steps (links are to commits in the PR that added the TIL section):

                                                                                                                                                                                                          1. Copied the PaperMod’s archives.html to layout/til/list.html to serve as the base list layout under /til/.
                                                                                                                                                                                                          2. Tweaked the new list.html to only show posts of type til, and removed counts and drafts.
                                                                                                                                                                                                          3. Added content/til/_index.md with the section’s title and description, which are shown at the top of the list view and in meta tags.
                                                                                                                                                                                                          4. Added my first TIL post with a lovely quote from How to Speak Whale.

                                                                                                                                                                                                          Unlike Jacob, I don’t mind TIL posts showing up in the main RSS feed. It’s enough for me that they don’t show up on the main page, which didn’t require any configuration tweaks in my case.

                                                                                                                                                                                                          When I push changes to the website, I like checking the gh-pages branch to see what got deployed. Recently I added an additional gh-pages-unminified branch to produce a more human-friendly diff. You can see the human-friendly result of merging the PR that added the TIL section here.

                                                                                                                                                                                                          That’s it for now. If I post TILs consistently, I will probably link to it from the top-level menu and tweak the /til/ page to also show posts by tag. For now, I like the idea of a quiet low-effort place to post publicly.

                                                                                                                                                                                                          Subscribe
                                                                                                                                                                                                            -
                                                                                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/07/23/using-yubikey-for-ssh-access/index.html b/til/2023/07/23/using-yubikey-for-ssh-access/index.html index f8627ee41..34a7815cd 100644 --- a/til/2023/07/23/using-yubikey-for-ssh-access/index.html +++ b/til/2023/07/23/using-yubikey-for-ssh-access/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                            Home » TIL: Today I learned...

                                                                                                                                                                                                            Using YubiKey for SSH access

                                                                                                                                                                                                            July 23, 2023

                                                                                                                                                                                                            I’ve been getting increasingly paranoid about computer security over the years. There’s always plenty to improve as new threats and technologies keep evolving. For example, dependency chain compromises like the one disclosed by PyTorch in December 2022 show that even a seemingly-benign action like installing a well-known package can result in exposure of secrets, including private SSH keys.

                                                                                                                                                                                                            As part of improving my security stance, I bought a couple of YubiKeys a couple of years ago and started using them wherever possible (either directly on supported sites or indirectly via Yubico Authenticator). At some point, I realised that YubiKeys may be used for SSH access as well, but I only got around to setting it up today. Turns out it’s pretty simple.

                                                                                                                                                                                                            One problem with YubiKeys is that they often offer more than one way of doing things, and SSH access is no exception. Andrej Friesen covered the options for SSH in an accessible post, which led me to the official Yubico page on securing SSH with FIDO2. I went with the discoverable key option along with ed25519 as the algorithm (see this post for a short explanation of the different algorithms).

                                                                                                                                                                                                            Following the official guide was straightforward, but then I hit this error: sign_and_send_pubkey: signing failed for ED25519-SK "..." from agent: agent refused operation. The common cause of this error is having the wrong permissions on the private key file, but that wasn’t the case for me. After a bit of digging, I found this Reddit thread that points to a gnome-keyring issue as the root problem (I use Ubuntu 22.04 on my laptop). As suggested on the Reddit thread, adding IdentityAgent none to the relevant hosts in my SSH config sorted out the issue. It seems like a reasonable workaround for now – I’m happy that I now have my SSH access tied to my YubiKey.

                                                                                                                                                                                                            Edit: Connection multiplexing can be useful for reducing the need to re-authenticate when running parallel connections.

                                                                                                                                                                                                            Subscribe
                                                                                                                                                                                                              -
                                                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/index.html b/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/index.html index 86a65491a..df4588b7a 100644 --- a/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/index.html +++ b/til/2023/08/11/the-rule-of-thirds-can-probably-be-ignored/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                              Home » TIL: Today I learned...

                                                                                                                                                                                                              The rule of thirds can probably be ignored

                                                                                                                                                                                                              August 11, 2023

                                                                                                                                                                                                              I recently read Cole Nussbaumer Knaflic’s Storytelling with You. Building on the success of Cole’s previous books on data visualisation, Storytelling with You contains detailed strategies on how to plan, create, and deliver compelling presentations. As I have a talk coming up at DataEngBytes Brisbane, I figured it was worth reading to sharpen my presentation skills.

                                                                                                                                                                                                              While the book contains many tips that I am already familiar with (e.g., overloading every slide with dense bullet points is not a good idea), one thing that was new to me was the rule of thirds for composing images. As described on Wikipedia, the rule of thirds “proposes that an image should be imagined as divided into nine equal parts by two equally spaced horizontal lines and two equally spaced vertical lines, and that important compositional elements should be placed along these lines or their intersections.”

                                                                                                                                                                                                              It’s nice to have rules to follow, but it seems like the rule of thirds was pretty much made up over two hundred years ago (see the Wikipedia article for historical details). Indeed, a recent study found that “for photographs that were rated as highly aesthetic and for a large set of paintings, calculated ROT [rule-of-thirds] values were about as low as in photographs that did not follow the rule of thirds.” And that “the rule of thirds seems to play only a minor, if any, role in large sets of high-quality photographs and paintings.” Similarly, an article from Adobe also states that the “rule” doesn’t have to be followed for a photo to be successful.

                                                                                                                                                                                                              That said, Cole notes that when using an image that fills the slide, following the rule of thirds leaves a bit more space for overlaying text. For example, I accidentally benefited from using a stock photo that followed the rule back when I built a recommender system for music from Bandcamp (see the cover photo there for a partial snapshot). In general, it’s worth being mindful about the composition of slides and other visual elements, which is where knowledge of rules of thumb can be useful. But when it comes to things that can be tested, like high-traffic websites, rigorously experimenting with positioning and composition may be the best approach.

                                                                                                                                                                                                              Subscribe
                                                                                                                                                                                                                -
                                                                                                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/08/14/email-notifications-on-public-github-commits/index.html b/til/2023/08/14/email-notifications-on-public-github-commits/index.html index 24ed002db..f2373293d 100644 --- a/til/2023/08/14/email-notifications-on-public-github-commits/index.html +++ b/til/2023/08/14/email-notifications-on-public-github-commits/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                Home » TIL: Today I learned...

                                                                                                                                                                                                                Email notifications on public GitHub commits

                                                                                                                                                                                                                August 14, 2023

                                                                                                                                                                                                                I have a daily data processing workflow that runs as a GitHub Actions cron job on the rls-data repository. The workflow turns some public Reef Life Survey data into JSONs that are used by tools on the Reef Life Survey website. Given that the repository is public, running the workflow is free.

                                                                                                                                                                                                                When I first implemented the workflow, it was running once a week, so I used GitHub’s settings to get notified every time any Actions workflow ran. This included successful runs, which wasn’t too noisy. However, I received many other emails due to Actions that ran on other projects I was working on.

                                                                                                                                                                                                                It was on my list to figure out a better way, but I only got around to it when I increased the flow’s frequency to daily. I still wanted to receive an email if the flow failed or made changes to the output JSONs, but there was no need for an email if it ran successfully without committing any changes.

                                                                                                                                                                                                                While GitHub supports getting emails on new pull requests by watching a repository, it doesn’t appear to support subscribing to commits. Fortunately, Stack Overflow has the answer for watching commits: Use the built-in RSS feed by appending /commits/<branch>.atom to the repository’s URL. In rls-data’s case, it is https://github.com/yanirs/rls-data/commits/master.atom, which I turned to emails using Blogtrottr.

                                                                                                                                                                                                                Problem solved! Now I only get emails on commits via Blogtrottr, along with emails on workflow failures directly from GitHub.

                                                                                                                                                                                                                Subscribe
                                                                                                                                                                                                                  -
                                                                                                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/index.html b/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/index.html index 835c2b944..e33a889e2 100644 --- a/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/index.html +++ b/til/2023/08/16/revisiting-start-small-stay-small-in-2023-chapter-1/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                  Home » TIL: Today I learned...

                                                                                                                                                                                                                  Revisiting Start Small, Stay Small in 2023 (Chapter 1)

                                                                                                                                                                                                                  August 16, 2023

                                                                                                                                                                                                                  I first read Start Small, Stay Small by Rob Walling in 2014, as I was working on a self-funded lifestyle business idea (aka micropreneurship). I ultimately abandoned the product I was working on in favour of salaried work, but now I’m considering new ideas, especially in the climate and nature-positive space.

                                                                                                                                                                                                                  As I spent the intervening years as an employee with VC-funded startups and with Automattic, my micropreneurship skills are much weaker than they could have been if I had stuck to building & selling products independently. Still, salaried work has its perks – I’m now in a more financially secure position than I was in 2014.

                                                                                                                                                                                                                  As part of getting back into micropreneurship, I figured it’d be worth rereading / skimming Start Small, Stay Small. This time, I’m using my TIL format to post some notes for my own reference.

                                                                                                                                                                                                                  I originally thought this would be a single post on the entire book, but my summary and thoughts on Chapter 1 (The Chasm Between Developer and Entrepreneur) ended up on the lengthy side. My goal with the TIL format is to publish more frequently, so this is good enough for a standalone post.

                                                                                                                                                                                                                  Summary:

                                                                                                                                                                                                                  • The book is focused on micropreneurs (solo founders who want to remain solo) and bootstrappers (those who want to grow a business without taking external funding) – both follow a similar process.
                                                                                                                                                                                                                  • Seeking funding takes a lot of time and focus, and limits the markets you can pursue. It’s harder to justify aiming for niche markets and moderate success when you’re funded by venture capitalists who expect a substantial return on investment.
                                                                                                                                                                                                                  • Defining a self-funded startup entrepreneur:
                                                                                                                                                                                                                    1. “technical visionary who creates software for a niche market”
                                                                                                                                                                                                                    2. “merges existing technical knowledge with online marketing knowledge”
                                                                                                                                                                                                                    3. “a cross between a developer, a webmaster, and a marketer”
                                                                                                                                                                                                                  • Wrong reasons to start:
                                                                                                                                                                                                                    1. “having a product idea” – “without a market, a software application is just a project”
                                                                                                                                                                                                                    2. “to get rich”
                                                                                                                                                                                                                    3. “because it sounds like fun”
                                                                                                                                                                                                                  • The right reasons to start depend on your goals. Micropreneurs lean towards lifestyle choices (freedom & income/location-independence). Bootstrappers might lean more towards the challenge and excitement of ownership and control. It’s worth spending time clarifying goals. Communicating them publicly and creating an accountability system is helpful in following through.
                                                                                                                                                                                                                  • Suggested goal: Build a startup that generates a monthly profit of $500. It’s harder than it may sound.
                                                                                                                                                                                                                  • Goals are key to getting through “the dip”: The point where the work is so hard (e.g., high volume of support) it can become unbearable.
                                                                                                                                                                                                                  • Reasons why people switch from development to entrepreneurship: lack of learning (keeping up with new tech becomes less exciting) and wanting more ownership (when working for a salary, little equity is retained if you leave).
                                                                                                                                                                                                                  • Roadblocks to success and how to avoid them:
                                                                                                                                                                                                                    1. “No market” – “building something no one wants”. Avoided by verifying there’s a market before building the product.
                                                                                                                                                                                                                    2. “Fear”. Can’t be fully avoided, but this may help: “The up-front fear is a big indicator that you’re going to grow as a person if you proceed through it. And, frankly, the terror wears off pretty quickly.”
                                                                                                                                                                                                                    3. “Lack of goals”, e.g., around profit growth and lifestyle. Avoided by defining your goals and writing them down.
                                                                                                                                                                                                                    4. “Inconsistency”, doing pseudo-productive things such as reading business books – can’t consume information and produce at the same time. Avoided by setting limits on content consumption – asking yourself whether pseudo-productive activities are actually worth it.
                                                                                                                                                                                                                    5. “Believing you have to do everything yourself”. Avoided by getting comfortable with outsourcing the right tasks to contractors and virtual assistants (e.g., probably outsource graphic design but don’t outsource the product architecture).
                                                                                                                                                                                                                  • Putting a dollar value on your work hours (i.e., dollarising your time) makes outsourcing decisions easier. It’s a step many entrepreneurs skip. This results in them performing menial tasks that can be outsourced, with an effective hourly rate that’s around the minimum wage or lower.
                                                                                                                                                                                                                  • Approaches to setting your current dollar value: (1) use freelancer rates; or (2) divide total compensation (including benefits) by work hours. Then set a desired rate. Don’t accept making something like $25 / hour. Make your target rate a reality as soon as possible, then increase it.
                                                                                                                                                                                                                  • Realisations that come from dollarising your time:
                                                                                                                                                                                                                    1. “Outsourcing is a bargain”.
                                                                                                                                                                                                                    2. “Keep work and play separate” – “work hard and play hard, but never do both at once”. Don’t do things like playing with your kids while working on your iPhone, as you’ll be doing both poorly.
                                                                                                                                                                                                                    3. “Wasting time is bad” – unproductive non-leisure activities are wasted money.
                                                                                                                                                                                                                    4. “Information consumption is only good when it produces something” (excluding consumption for leisure). Recommendation: “When reading blogs or books or listening to podcasts or audio books, take action notes.” If no actions arise, it may be that the content is low value.
                                                                                                                                                                                                                  • Realisations that come when transitioning from developer to entrepreneur:
                                                                                                                                                                                                                    1. “Being a good technician is not enough”. It’s critical to do management work like thinking about return on investment and productivity, and visionary/creative work around the long-term direction of the business. This is a key component in escaping the $25 / hour pit.
                                                                                                                                                                                                                    2. “Market comes first, marketing second, aesthetic third, and functionality a distant fourth”.
                                                                                                                                                                                                                    3. “Things will never be as clear as you want them to be” – writing code is straightforward in comparison to the ever-changing market, which requires a lot of experimentation to get right.
                                                                                                                                                                                                                    4. “You can’t specify everything, but you do need a plan”.
                                                                                                                                                                                                                    5. “You need to fail fast and recover”.
                                                                                                                                                                                                                    6. “You will never be done” – building and then collecting money is a pipe dream; product & marketing require continuous investment to remain successful.
                                                                                                                                                                                                                    7. “Don’t expect instant gratification” – product/marketing/reputation require time and effort, and it’s way harder the first time. The real work begins after you launch.
                                                                                                                                                                                                                    8. “Process is king” – having documented repeatable processes is key to delegation, bringing on partners, and avoiding mistakes. Such documentation makes it easier to sell the business if you want.
                                                                                                                                                                                                                    9. “Nothing about a startup is a one-time effort” – getting to the point of an automated startup requires a wise choice of niche & product, as well as investment in outsourcing and automation. Things like marketing remain hard to outsource, though.

                                                                                                                                                                                                                  Key quotes:

                                                                                                                                                                                                                  • “A developer who knows how to market a product is a rare (and powerful) combination.”
                                                                                                                                                                                                                  • “Marketing is more important than your product. […] Product Last. Marketing First.”
                                                                                                                                                                                                                  • “If you’re a venture-backed startup founder you’re looking at many years of long hours with a small potential for a huge payoff. […] If you’re a self-funded startup founder, you’re looking at a decent potential for a decent payoff.”
                                                                                                                                                                                                                  • “Without a market, a software application is just a project.”

                                                                                                                                                                                                                  My thoughts:

                                                                                                                                                                                                                  • I’m surprised by the length of my summary! I thought it’d just be a couple of quotes, especially given that many of the specific examples and references haven’t aged well. But a lot of the key principles are still relevant today.
                                                                                                                                                                                                                  • It’s somewhat ironic that reading business books is described as non-productive given that Start Small, Stay Small is a business book, but I suppose that’s qualified by the later statement that information consumption is worthwhile when it leads to productive action notes. As I enjoy reading & learning, I can definitely relate to the sense of pseudo-productivity when going down information rabbit holes. Further, since the book was published, the number of ways to get distracted has kept increasing while the number of hours in a day hasn’t changed, so remaining focused is perhaps more of a challenge these days.
                                                                                                                                                                                                                  • Walling talks about not being able to get rich through salaried work (in the context of the desire to get rich being a wrong reason to start), but I disagree. It’s well-known that many tech employees earn well, even outside the big tech companies (where total compensation can be in the high six figures or even in the seven figures). Working consistently for a salary and keeping expenses under control is a safe way to get rich, but it can be hard (see the FIRE movement). In my case, I would have been richer now if I had joined Google after my PhD in 2012 (I interned there and chose to work with small startups instead), or if I had not done a PhD and stayed in Israel to work with big tech companies in 2009 (tech compensation in Israel is higher than in Australia), or if I had stayed with Automattic a couple of years ago and kept working full time. But life isn’t only about maximising material wealth – I’m happy with my choices.
                                                                                                                                                                                                                  • There are multiple references to long nights, which I assume mostly apply to people who work on a side-business in addition to a full-time job. I suppose there’s no avoiding some unpleasant work at inconvenient times, but applying time discipline is important, especially if control over how you spend your time is a motivator for going down the micropreneurship path.
                                                                                                                                                                                                                  • One segment that feels dated in 2023 discusses examples of tasks that can be easily outsourced, like one-off scraping of images from a website and making some CSS tweaks. These days, it’s cheaper & faster to prompt ChatGPT or one of its cousins to get such tasks done. Echoing Michael Lynch’s review of the book, I also have my doubts about Walling’s advice on outsourcing, but there are definitely tasks that can and should be outsourced.
                                                                                                                                                                                                                  • I remembered the book as being too militant about dollarising time. The simple fact is that time can’t be saved like money (e.g., when you die your heirs don’t get to enjoy all the time you’ve saved). However, at least in chapter 1, it’s clear that dollarising time refers to work time – leisure time is a different story. It makes perfect sense to be diligent about how work time is spent and aim to at least match the market value of your time when running a for-profit business. Still, it’s hard to put a price on joy and purpose found in work – many people are happy to take a pay cut for more fulfilling work. Fulfillment isn’t captured by the crude metric of dollars earned per hour worked. Nonetheless, at least being aware of your effective hourly rate as a micropreneur seems like a good guardrail.
                                                                                                                                                                                                                  • To dollarise time, you need to know how many hours you work. But the reality is that we use the same brain for work and outside work, and total control over thoughts is impossible. I doubt it’s even desirable, as many good ideas come when we’re not “at work”. Still, it’s worth striving for some separation between work and non-work time, especially given the ubiquity of internet connections and mobile devices.
                                                                                                                                                                                                                  • On this read, I found the paragraphs that talked about boredom with keeping up with tech especially relatable. While there have been transformative changes in tech over the past decade, anything useful quickly becomes a software commodity – less exciting from a technical perspective. Everyday tech work often requires putting the right lego pieces together and dealing with human problems. While I still find some satisfaction in the technical aspects of building software and exploring data, I’m less interested in tech for tech’s sake these days.
                                                                                                                                                                                                                  • I abandoned my last micropreneurship attempt because I came to the same realisation around the real work beginning after launch. I just didn’t care enough about my online price comparison product to put in months and years of effort into marketing. This is partly what’s missing from chapter 1 (though I suppose it’s somewhat covered by setting goals): For some people, aligning business value with personal values is key to putting in consistent effort. It’s easy to give up when there’s more money and less stress in salaried work, and you don’t care about the product you’ve built. If I decide to follow the micropreneur path again, one of my aims is to have better value alignment than last time.

                                                                                                                                                                                                                  Key action item from this chapter: Think deeply on goals and commit to them.

                                                                                                                                                                                                                  Subscribe
                                                                                                                                                                                                                    -
                                                                                                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/index.html b/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/index.html index c945e9de4..49b4eecd2 100644 --- a/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/index.html +++ b/til/2023/08/17/revisiting-start-small-stay-small-in-2023-chapter-2/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                    Home » TIL: Today I learned...

                                                                                                                                                                                                                    Revisiting Start Small, Stay Small in 2023 (Chapter 2)

                                                                                                                                                                                                                    August 17, 2023

                                                                                                                                                                                                                    Following my previous TIL post on Chapter 1 of Start Small, Stay Small by Rob Walling, this post covers Chapter 2: Why Niches Are the Name of the Game.

                                                                                                                                                                                                                    Summary:

                                                                                                                                                                                                                    • Reiterating the need to find a market before building a product, which is the opposite of many developers’ approach of building a product first.
                                                                                                                                                                                                                    • If you get lucky, you can find success with building the product first, but the odds aren’t in your favour. It’s a similar story for many VC-backed startups.
                                                                                                                                                                                                                    • Reasons why people still follow the VC-backed startup: The lottery factor of a massive success / cashing out, and potential personal popularity. If the reason is cashing out, it’s worth thinking what you’d do later. If the goal is to work on projects you enjoy, you can start now without taking VC funding.
                                                                                                                                                                                                                    • Reasons you must go niche:
                                                                                                                                                                                                                      1. “A niche requires you to narrow your product focus”. For example, you can build the perfect product for a single person you know well. Expanding this to a group of people that you keep happy leads to guaranteed revenue.
                                                                                                                                                                                                                      2. “Niche advertising is more cost-effective”.
                                                                                                                                                                                                                      3. “Niches have less competition”. Big companies don’t bother with products for small markets, e.g., Microsoft wouldn’t build a product for a market that generates an annual revenue of $500,000.
                                                                                                                                                                                                                      4. “Niches have higher profit margins”. This is a result of having less competition.
                                                                                                                                                                                                                      5. “Niche markets are not used to good marketing”.
                                                                                                                                                                                                                      6. “It’s easier for prospects to trust you”. This is because they’re more likely to hear about you multiple times (related to the cost-effectiveness of advertising).
                                                                                                                                                                                                                    • Warm niches exercise: Write up a table with two columns. Then fill it up with names of people you know (column 1) and their work experiences / hobbies (column 2).
                                                                                                                                                                                                                    • Common sentiment: All good niches are already taken. People starting out want an indication that the chosen niche is going to work, but it requires a leap of faith. Quoting previous chapter: “things will never be as clear as you want them to be”.
                                                                                                                                                                                                                    • Approaches to brainstorming niches (warm niches are best to increase your chances of success, but some cold niches can be made warm through networking):
                                                                                                                                                                                                                      1. “Look at all areas of your life”.
                                                                                                                                                                                                                      2. “Look at occupations”.
                                                                                                                                                                                                                      3. “Cheat” by going through lists of existing ideas (the book includes many links such as A Startup A Day, where the last post is from March 2011).
                                                                                                                                                                                                                    • Evaluating a niche:
                                                                                                                                                                                                                      • Focus on consumers and/or small businesses as they have purchasing authority, make decisions fast, and search for online solutions (not the case for large companies and government agencies, for example).
                                                                                                                                                                                                                      • Check that the market is large enough. Rule of thumb: can place an ad in a specific magazine for less than $5,000 (though offline magazine advertising isn’t recommended, and the inexistence of a magazine doesn’t imply that the market isn’t viable). Can also check labour statistics for the number of practitioners in a field. Be wary of markets that don’t have dedicated websites or magazines, and where labour statistics claim less than 10,000 members.
                                                                                                                                                                                                                      • Check that there’s an inexpensive way to reach customers (typically online) – your niche is unlikely to be people who visit tech sites. The focus should be on where your customers are (e.g., a niche website). Top approaches for micropreneurs: building an audience and search engine optimisation. Secondary approaches include referral traffic, partnerships, article marketing, and cold calling. It’s hard to generate a sustainable stream of prospects without the top approaches.
                                                                                                                                                                                                                    • It’s better to focus on vertical markets (e.g., single industry or hobby) than on horizontal markets because members of a vertical have similar behaviours, talk to each other, hang out together, and have similar needs. This makes them easier to target, and increases the chances of organic product growth. Horizontal markets are rarely a good idea for micropreneurs because they’re too large and expensive to navigate.
                                                                                                                                                                                                                    • Measuring market demand without spending money:
                                                                                                                                                                                                                      • Obtain likely conversion rates for your price point.
                                                                                                                                                                                                                      • Discover likely traffic volumes through keyword research tools (considering search engine traffic, incoming links, direct traffic, and advertising).
                                                                                                                                                                                                                      • Example: 5,000 people search for inventory software each month. If you rank #1 for the term and get another 5,000 visitors from other sources, with a 0.5% conversion rate and a $200 product, you’d make $10,000 per month.
                                                                                                                                                                                                                      • Exercise: Take the top five hobbies and occupations from the warm niche brainstorming, ranked by personal interest. Ask the person you know about their problems to uncover software needs and ideas. Then use keyword research tools to see if there are other ideas that the person didn’t mention.
                                                                                                                                                                                                                      • Measure demand using free keyword research tools. There are also paid tools that can help, especially with assessing the difficulty of ranking for a term (many free tools are junk). As a rule of thumb, multiply by four the number of searches for generic terms like attorney billing software to get an estimate of traffic if you rank #1 for the generic term (due to traffic from long tail terms and other sources of traffic).
                                                                                                                                                                                                                      • Check the competition – regardless of what the tools say, if the current top result is well-optimised and has a high PageRank, it’s going to be tough to beat.
                                                                                                                                                                                                                    • Testing an idea for less than $100 (works for SaaS but not for products that rely on network effects):
                                                                                                                                                                                                                      1. Choose the most interesting idea from your shortlist of product options (following niche & market demand research).
                                                                                                                                                                                                                      2. Set up a mini sales site with 2-3 pages (homepage, pricing / signup, and possibly a product tour).
                                                                                                                                                                                                                      3. Try to get people to click a “buy now” or “free trial” button (depending on the product cost).
                                                                                                                                                                                                                      4. Create an AdWords (now Google Ads) campaign to generate traffic.
                                                                                                                                                                                                                      5. Track clicks to estimate the conversion rate and link it to keywords. Notify users who are interested that the product is still under development.
                                                                                                                                                                                                                      6. If you feel bad about misleading potential customers, include a “Coming Soon” note somewhere.

                                                                                                                                                                                                                    Key quotes:

                                                                                                                                                                                                                    • “The product with a sizable market and low competition wins even with bad marketing, a bad aesthetic, and poor functionality.”
                                                                                                                                                                                                                    • “With luck on your side you don’t need money, good marketing or a solid product. You just need to be lucky.”
                                                                                                                                                                                                                    • “What matters is finding a group of people who need your something more than they need the money you’re charging for it.”
                                                                                                                                                                                                                    • “The best niches are reserved for people who do something.”
                                                                                                                                                                                                                    • “As a self-funded startup you want a market that is already looking for your product, even if it doesn’t exist. This is because creating demand is very, very expensive while filling existing demand is, by comparison, cheap.”
                                                                                                                                                                                                                    • “If your target market is not online, you have no chance of succeeding using the methodologies you’ll find in this book. This is non-negotiable.”
                                                                                                                                                                                                                    • “When you receive 50,000 visitors from one of the major media sites you will be lucky to convert five sales.”
                                                                                                                                                                                                                    • “Unfortunately, great products are often built and launched without a thought given to how the target audience will find out about it. You must have an inexpensive, ongoing source of new customers.”
                                                                                                                                                                                                                    • “With niche research the problem is not finding new ideas, but narrowing to the most effective strategies that you can implement in a reasonable amount of time.”
                                                                                                                                                                                                                    • “You only need to master two skills to sell online: human behavior and math.”

                                                                                                                                                                                                                    My thoughts:

                                                                                                                                                                                                                    • Despite agreeing with the overall message of focusing on niche markets, I find myself thinking of counter-examples (e.g., of people who got lucky playing the startup or product-first games). I suppose it’s similar to the note towards the end of Thinking, Fast and Slow – being aware of biases isn’t enough to eliminate them. Base rate neglect is one such bias.
                                                                                                                                                                                                                    • Much of the discussion around market size and the ability to reach customers reminded me of Jason Cohen’s post on the difference between successfully solving a problem and having a viable business model. As the post came out in 2023, it’s likely that people are still making the same mistakes, which is related to it being hard to fight our biases.
                                                                                                                                                                                                                    • I don’t fully agree with Walling’s note on choosing the micropreneur path as a way of working on enjoyable projects. Starting a business isn’t a good way to guarantee work on things you enjoy, unless you enjoy everything that comes with running a business. That is, I don’t believe that running a VC-funded startup is that different from running a bootstrapped startup when it comes to enjoyment – there will always be unpleasant tasks. In line with my post on Chapter 1, I do believe that with a bootstrapped startup the founders have more control over aligning the work with their values – it’s hard to say no to investors who are essentially your bosses. Theoretically, with a bootstrapped startup it’s easier to say no to certain business activities and forgo the potential market share that comes with them, if pursuing such activities disagrees with your values – micropreneurs don’t have to pursue growth at all costs.
                                                                                                                                                                                                                    • The book includes outdated references to magazine & newspaper ads, but it’s easy to mentally translate the concepts to today’s tech. I suppose the equivalents today are niche sites / newsletters / online magazines / influencer channels where one can advertise. The underlying principles age slower than specific technologies and tools.
                                                                                                                                                                                                                    • A trap I hit the last time I read the book in 2014 is having a product idea and then insisting that a niche exists to match the product idea. It’s probably easier to start without a product idea, as we tend to fall in love with our ideas.
                                                                                                                                                                                                                    • Mentions of Web 2.0 as a shiny new thing bring back memories, and show how little has changed conceptually. The recent hype around blockchain and web3 looked a lot like solutions looking for problems to me.
                                                                                                                                                                                                                    • It’s hard to learn from the experience and advice of others. Between 2014 and now, I’ve seen multiple examples of failures that are due to not following (or being aware of) key advice from the book. Still, I’m somewhat swayed by misleading media noise and specific founder stories.
                                                                                                                                                                                                                    • I’m not in the marketing world, so I’m curious to what extent things have changed around specific tooling & approaches. Walling’s newer books might help (and there are a million and one others). But again, principles are key – just like in data science and software engineering where new tools often re-implement old ideas.
                                                                                                                                                                                                                    • Similarly, I’m curious to what extent website traffic is important these days, as websites aren’t the only way to reach people online (e.g., there are app stores and social media channels). However, thinking about impressions and conversion rates is important regardless of the medium through which they’re obtained.
                                                                                                                                                                                                                    • The specifics of market demand research seem outdated. It’s probably better to rely on newer focused resources for specifics, though people still use search engines. Free keyword research tools still exist in 2023 and are probably a good way to get started. Following the exact tips from the book too closely would be silly, though.
                                                                                                                                                                                                                    • A mini sales site makes perfect sense, and is probably still cheap to build and test. It may even be cheaper today given the abundance of templates and tools to build static websites. Further, static sites can be hosted for free, with the only cost being the domain name (that’s the case for my site).

                                                                                                                                                                                                                    Key action items from this chapter:

                                                                                                                                                                                                                    • Do the warm niche exercise
                                                                                                                                                                                                                    • Find niches worth targeting (warmness is important, but alignment with interests and values is key)
                                                                                                                                                                                                                    • Evaluate niche market size
                                                                                                                                                                                                                    • Figure out how to reach promising niches
                                                                                                                                                                                                                    • Interview people in the promising niches
                                                                                                                                                                                                                    • Measure market demand for specific ideas
                                                                                                                                                                                                                    • Narrow down the list of ideas
                                                                                                                                                                                                                    • Build and market a mini sales site to gauge feasibility

                                                                                                                                                                                                                    That’s plenty of work, so I probably won’t get to the next chapter for a while!

                                                                                                                                                                                                                    Subscribe
                                                                                                                                                                                                                      -
                                                                                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/index.html b/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/index.html index e3857c407..c4493c4d4 100644 --- a/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/index.html +++ b/til/2023/08/21/the-minimalist-entrepreneur-is-too-prescriptive-for-me/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                      Home » TIL: Today I learned...

                                                                                                                                                                                                                      The Minimalist Entrepreneur is too prescriptive for me

                                                                                                                                                                                                                      August 21, 2023

                                                                                                                                                                                                                      I picked up The Minimalist Entrepreneur by Sahil Lavingia from the local library after coming across Gumroad a few times and getting intrigued by its creation story. I was hoping for general but useful advice on starting a bootstrapped business, but was disappointed by the overly prescriptive nature of the book. It feels like the author over-generalised from his own experience, e.g., in emphasising organic community-based growth over other marketing channels, and in insisting that one must share the story of building the business on social media. Example quote: “people don’t care about your business and its success, they care about you and your struggles” – I’d say that most people don’t care about either you or your business, but I can see how Sahil’s story is interesting to Gumroad’s users (who are also small business owners / creators). There are countless examples of products I use where I know nothing about the people who make them. I use those products because they address a need or a want.

                                                                                                                                                                                                                      The book was off to a good start with Sahil’s story and advice that is in line with advice from Start Small, Stay Small, which I revisited recently. However, beyond around the midpoint of the book, I started losing patience with its prescriptiveness and began speed-reading and skipping irrelevant stories. As one Goodreads reviewer said, the book felt like “three blog posts in a trench coat”, which I suppose is unsurprising as it was born out of a blog post.

                                                                                                                                                                                                                      Given that it was a quick weekend read, the book wasn’t a complete waste of time. It probably would have been better if it remained focused on Sahil’s own story, which I found the most engaging. The best advice is perhaps in the intro: “You definitely do not need to finish this book to start. […] You start, then learn”.

                                                                                                                                                                                                                      Subscribe
                                                                                                                                                                                                                        -
                                                                                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/index.html b/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/index.html index 0ebce7068..0132aad1b 100644 --- a/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/index.html +++ b/til/2023/09/21/googles-rules-of-machine-learning-still-apply-in-the-age-of-large-language-models/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                        Home » TIL: Today I learned...

                                                                                                                                                                                                                        Google's Rules of Machine Learning still apply in the age of large language models

                                                                                                                                                                                                                        September 21, 2023

                                                                                                                                                                                                                        I heard about Google’s Rules of Machine Learning (ML) maybe 4-5 years ago. Much like Steve McConnell’s classic software engineering mistakes, the rules capture lessons learned from software engineering projects, though they are focused on the problems that arise from the engineering problem of shipping ML systems to production.

                                                                                                                                                                                                                        Despite the excitement about playing with data and models, the reality of building ML systems is that it’s mostly an engineering problem. This remains the case in the age of large language models. Perhaps it’s even more so because integrating language models into a product can be as simple as calling an API, which should make it easier to focus on business problems, pipelines, data, and evaluation. It’s important to remember that at an abstract level, ML is just a data transformation – there is no magic involved.

                                                                                                                                                                                                                        As the page containing Google’s ML rules is long and detailed, I put together this TIL post for my own ease of reference. It contains the key quote from the overview, along with the rules without their explanations. Go to the source for further details.

                                                                                                                                                                                                                        Overview

                                                                                                                                                                                                                        To make great products:

                                                                                                                                                                                                                        do machine learning like the great engineer you are, not like the great machine learning expert you aren’t.

                                                                                                                                                                                                                        Most of the problems you will face are, in fact, engineering problems. Even with all the resources of a great machine learning expert, most of the gains come from great features, not great machine learning algorithms. So, the basic approach is:

                                                                                                                                                                                                                        1. Make sure your pipeline is solid end to end.
                                                                                                                                                                                                                        2. Start with a reasonable objective.
                                                                                                                                                                                                                        3. Add common-sense features in a simple way.
                                                                                                                                                                                                                        4. Make sure that your pipeline stays solid.

                                                                                                                                                                                                                        The Rules

                                                                                                                                                                                                                        Before Machine Learning

                                                                                                                                                                                                                        1. Don’t be afraid to launch a product without machine learning.
                                                                                                                                                                                                                        2. First, design and implement metrics.
                                                                                                                                                                                                                        3. Choose machine learning over a complex heuristic.

                                                                                                                                                                                                                        ML Phase I: Your First Pipeline

                                                                                                                                                                                                                        1. Keep the first model simple and get the infrastructure right.
                                                                                                                                                                                                                        2. Test the infrastructure independently from the machine learning.
                                                                                                                                                                                                                        3. Be careful about dropped data when copying pipelines.
                                                                                                                                                                                                                        4. Turn heuristics into features, or handle them externally.

                                                                                                                                                                                                                        Monitoring

                                                                                                                                                                                                                        1. Know the freshness requirements of your system.
                                                                                                                                                                                                                        2. Detect problems before exporting models.
                                                                                                                                                                                                                        3. Watch for silent failures.
                                                                                                                                                                                                                        4. Give feature columns owners and documentation.

                                                                                                                                                                                                                        Your First Objective

                                                                                                                                                                                                                        1. Don’t overthink which objective you choose to directly optimize.
                                                                                                                                                                                                                        2. Choose a simple, observable and attributable metric for your first objective.
                                                                                                                                                                                                                        3. Starting with an interpretable model makes debugging easier.
                                                                                                                                                                                                                        4. Separate Spam Filtering and Quality Ranking in a Policy Layer.

                                                                                                                                                                                                                        ML Phase II: Feature Engineering

                                                                                                                                                                                                                        1. Plan to launch and iterate.
                                                                                                                                                                                                                        2. Start with directly observed and reported features as opposed to learned features.
                                                                                                                                                                                                                        3. Explore with features of content that generalize across contexts.
                                                                                                                                                                                                                        4. Use very specific features when you can.
                                                                                                                                                                                                                        5. Combine and modify existing features to create new features in human­-understandable ways.
                                                                                                                                                                                                                        6. The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have.
                                                                                                                                                                                                                        7. Clean up features you are no longer using.

                                                                                                                                                                                                                        Human Analysis of the System

                                                                                                                                                                                                                        1. You are not a typical end user.
                                                                                                                                                                                                                        2. Measure the delta between models.
                                                                                                                                                                                                                        3. When choosing models, utilitarian performance trumps predictive power.
                                                                                                                                                                                                                        4. Look for patterns in the measured errors, and create new features.
                                                                                                                                                                                                                        5. Try to quantify observed undesirable behavior.
                                                                                                                                                                                                                        6. Be aware that identical short-term behavior does not imply identical long-term behavior.

                                                                                                                                                                                                                        Training-Serving Skew

                                                                                                                                                                                                                        1. The best way to make sure that you train like you serve is to save the set of features used at serving time, and then pipe those features to a log to use them at training time.
                                                                                                                                                                                                                        2. Importance-weight sampled data, don’t arbitrarily drop it!
                                                                                                                                                                                                                        3. Beware that if you join data from a table at training and serving time, the data in the table may change.
                                                                                                                                                                                                                        4. Re-use code between your training pipeline and your serving pipeline whenever possible.
                                                                                                                                                                                                                        5. If you produce a model based on the data until January 5th, test the model on the data from January 6th and after.
                                                                                                                                                                                                                        6. In binary classification for filtering (such as spam detection or determining interesting emails), make small short-term sacrifices in performance for very clean data.
                                                                                                                                                                                                                        7. Beware of the inherent skew in ranking problems.
                                                                                                                                                                                                                        8. Avoid feedback loops with positional features.
                                                                                                                                                                                                                        9. Measure Training/Serving Skew.

                                                                                                                                                                                                                        ML Phase III: Slowed Growth, Optimization Refinement, and Complex Models

                                                                                                                                                                                                                        1. Don’t waste time on new features if unaligned objectives have become the issue.
                                                                                                                                                                                                                        2. Launch decisions are a proxy for long-term product goals.
                                                                                                                                                                                                                        3. Keep ensembles simple.
                                                                                                                                                                                                                        4. When performance plateaus, look for qualitatively new sources of information to add rather than refining existing signals.
                                                                                                                                                                                                                        5. Don’t expect diversity, personalization, or relevance to be as correlated with popularity as you think they are.
                                                                                                                                                                                                                        6. Your friends tend to be the same across different products. Your interests tend not to be.
                                                                                                                                                                                                                        Subscribe
                                                                                                                                                                                                                          -
                                                                                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/index.html b/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/index.html index 1b6a496c3..eeb1be152 100644 --- a/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/index.html +++ b/til/2023/09/25/the-lines-between-solo-consulting-and-product-building-are-blurry/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                          Home » TIL: Today I learned...

                                                                                                                                                                                                                          The lines between solo consulting and product building are blurry

                                                                                                                                                                                                                          September 25, 2023

                                                                                                                                                                                                                          I’ve been thinking about starting a small product business recently – something niche enough that I can build independently, while aiming for early profitability. As getting customers for any product requires marketing, I figured I could improve my marketing skills if I started by selling a product that already has a market: My services as a Data & AI Consultant. As the way I currently present myself is somewhat generic, my thinking was that this would be a good opportunity to improve my positioning by building dedicated landing pages for specific services and target audiences – something that’s necessary for any product.

                                                                                                                                                                                                                          I chatted a bit with Bing on keyword research and this is one of the things it said:

                                                                                                                                                                                                                          First of all, I think you should define your target audience and your value proposition clearly. Who are you trying to help with your consulting site? What problems can you solve for them? What benefits can they get from booking a call with you? These questions will help you craft a compelling message that attracts the right leads.

                                                                                                                                                                                                                          One of the resources it cited was a post on consulting best practices from Consulting Success, where some of the tips are around following proven processes, offering productised solutions, and developing a Magnetic Message, which they formulate as “I help [WHO] to [solve WHAT problem] so they can [see WHAT results]. My [WHY choose me]…” Another post by Consulting Success talks about building a consulting website, with the first step being to “understand your ideal client”.

                                                                                                                                                                                                                          This sort of advice might be obvious, but I suppose it’s needed because it’s easy to fall into the trap of making a consulting site about the consultant rather than about the ideal client and their problems. The parallel in the product world is starting from a product idea rather than from customer needs, which many people who enjoy building stuff tend to do.

                                                                                                                                                                                                                          The ideal bit is worth emphasising because going too broad can also be a mistake for independent consultants and product builders. Jonathan Stark addresses this point concisely, saying that “the only business strategy you’ll ever need [is to] help people you like get what they want.” This resonates with me since one of the reasons I want to work independently is to have choice over who I help – I find it hard to deal with the moral conundrums that come with working in a business that sells technology solutions to pretty much anyone (as long it’s legal).

                                                                                                                                                                                                                          Starting with decisions on a niche and ideal clients is in line with the advice to product-focused micropreneurs from Start Small, Stay Small. It makes sense: If one of the aims is to keep the headcount low (starting from one and perhaps staying there), it’s impossible to effectively target a broad market. This is regardless of whether the product sold is software subscriptions or consulting services. What wasn’t obvious to me until I came across Consulting Success and similar resources is how similar solo consulting can be to solo product building. I thought of consulting more as freelancing or contracting – selling time/effort for money. But thinking of consulting as achieving results and solving problems for clients brings it much closer to the product realm. Further, through consulting, it’s possible to uncover problems that are shared by many clients, which can become the basis for a pure self-service software solution.

                                                                                                                                                                                                                          Subscribe
                                                                                                                                                                                                                            -
                                                                                                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/index.html b/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/index.html index f2cd847e2..ff06ff745 100644 --- a/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/index.html +++ b/til/2023/10/06/artificial-intelligence-was-a-marketing-term-all-along-just-call-it-automation/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                            Home » TIL: Today I learned...

                                                                                                                                                                                                                            Artificial intelligence was a marketing term all along – just call it automation

                                                                                                                                                                                                                            October 6, 2023

                                                                                                                                                                                                                            Quoting Emily M. Bender:

                                                                                                                                                                                                                            What is AI?

                                                                                                                                                                                                                            In fact this is a marketing term. It’s a way to make certain kinds of automation sound sophisticated, powerful, or magical and as such it’s a way to dodge accountability by making the machines sound like autonomous thinking entities rather than tools that are created and used by people and companies. It’s also the name of a subfield of computer science concerned with making machines that “think like humans” but even there it was started as a marketing term in the 1950s to attract research funding to that field.

                                                                                                                                                                                                                            I think that discussions of this technology become much clearer when we replace the term AI with the word “automation”. Then we can ask:

                                                                                                                                                                                                                            • What is being automated?
                                                                                                                                                                                                                            • Who’s automating it and why?
                                                                                                                                                                                                                            • Who benefits from that automation?
                                                                                                                                                                                                                            • How well does the automation work in its use case that we’re considering?
                                                                                                                                                                                                                            • Who’s being harmed?
                                                                                                                                                                                                                            • Who has accountability for the functioning of the automated system?
                                                                                                                                                                                                                            • What existing regulations already apply to the activities where the automation is being used?

                                                                                                                                                                                                                            Marketing isn’t a bad thing – it all depends on what you market and the tactics you use. But getting to the bottom of things does require going beyond marketing lingo, which Bender does well in the above quote.

                                                                                                                                                                                                                            I was curious about her claim that AI started as a marketing term. After some searching, I got to the Wikipedia page on the 1956 Dartmouth Workshop, which says that:

                                                                                                                                                                                                                            In 1955, John McCarthy, then a young Assistant Professor of Mathematics at Dartmouth College, decided to organize a group to clarify and develop ideas about thinking machines. He picked the name ‘Artificial Intelligence’ for the new field. He chose the name partly for its neutrality; avoiding a focus on narrow automata theory, and avoiding cybernetics which was heavily focused on analog feedback, as well as him potentially having to accept the assertive Norbert Wiener as guru or having to argue with him.

                                                                                                                                                                                                                            I suppose that applying for research funding is a form of marketing, which is in line with Bender’s claim.

                                                                                                                                                                                                                            In any case, talking about automation of specific tasks rather than about “using AI” is a useful trick that I’ll be using in concrete discussions. Unfortunately, I’ll also keep using the AI marketing term where suitable, e.g., in my current title of Data & AI Consultant. After all, titles are a part of marketing.

                                                                                                                                                                                                                            Subscribe
                                                                                                                                                                                                                              -
                                                                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/index.html b/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/index.html index 508144171..b4406abf9 100644 --- a/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/index.html +++ b/til/2023/11/21/you-dont-need-a-proprietary-api-for-static-maps/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                              Home » TIL: Today I learned...

                                                                                                                                                                                                                              You don't need a proprietary API for static maps

                                                                                                                                                                                                                              November 21, 2023

                                                                                                                                                                                                                              In addition to my long-time volunteering as a scuba diver with Reef Life Survey (RLS), I’ve been helping them with bits and pieces around data engineering / science / web work (somewhat reluctantly). A few months ago, we discovered an issue in the PDF field guides exported from Reef Species of the World: species distribution maps embedded in the PDFs were broken. This was due to a change in Google’s Static Maps API, which became a paid feature.

                                                                                                                                                                                                                              While paying for the Google API would have been the simplest solution (and fairly cheap with appropriate caching), this was an opportunity to improve the static map functionality while removing the paid proprietary API dependency. For me, it was also an opportunity to learn a bit about geospatial analysis in Python, which I’ve been curious about.

                                                                                                                                                                                                                              A bit more context on the problem: Each of the species in the PDF field guides is shown with its distribution map, as recorded in the RLS dataset. Some species are widespread and common, like Labroides dimidatus (a cleaner wrasse that was recorded in over 4,000 sites). Unlike the maps presented on the web version, PDF maps are meant to fit a box that’s about 4.5cm by 3.5cm when printed, so space is limited for on-map labels.

                                                                                                                                                                                                                              An important limitation of the Google Static Maps API (which is shared by the cheaper Mapbox API) is that of request URL length. This isn’t an issue for maps with a few custom features, but requesting a map with thousands of markers isn’t feasible without reducing coordinate accuracy and clustering markers to reduce their number. This complicates the code that calls the static mapping API, and can easily lead to unexpected results, like fish found on dry land.

                                                                                                                                                                                                                              I suppose that an attractive feature of the Google Static Maps API is the simplicity of embedding maps in pure front-end applications, as it obviates the need to implement a back-end to generate the static maps. However, this feature was irrelevant to the PDF generation task, which happens on the back-end anyway.

                                                                                                                                                                                                                              Once I understood the downsides of sticking with proprietary static map APIs (including their limited customisability), I realised I could expand the Python data processing code in the rls-data repo to pre-generate all the maps whenever new survey data becomes available. The final result was about 5,000 distribution maps that are committed to the repo. This admittedly stretches the common use cases for Git repos, but at about 15KB per map, it’s not terrible. In any case, it’d be easy to store the maps on S3 if needed.

                                                                                                                                                                                                                              The full code with the change, including the GitHub Action that refreshes the maps, is in this PR. It’s a bit hard to navigate since GitHub doesn’t like PRs with thousands of files, but the commit history gives the full picture of my experimentation with Python-based mapping solutions. The map generation code that ended up getting merged starts here.

                                                                                                                                                                                                                              Python has a large ecosystem of geospatial packages, so choosing the right packages for the use case was a bit tricky. However, I heard about geopandas, so I used it for my first round of experiments. I got reasonable-looking maps, but it was a bit slow. I also found the auto-zoom functionality frustrating – given the space constraints, balancing the zoom level with keeping a constant aspect ratio and the need for legibility seemed non-trivial (at least to me).

                                                                                                                                                                                                                              Some discussions on zooming with ChatGPT led to it mentioning cartopy. I was quickly sold on it given all the pretty maps in the cartopy gallery. It also turned out to be much faster – generating the maps with cached tiles (using geopandas and contextily) was six times slower than using cartopy with Natural Earth features. The cartopy solution was also twice as fast as using geopandas with Natural Earth, and I could easily set the colour of the ocean to match the colour used on the RLS website – definitely a winner! A full run to regenerate all the maps with a standard GitHub Actions runner takes about 3.5 minutes, which is reasonable for something that runs at most daily.

                                                                                                                                                                                                                              I’m far from a geospatial expert, so the solution I landed on for zooming with a constant aspect ratio isn’t great: There are a few hard-coded map areas with recognisable coastlines (Australia, Europe, North America, etc.), which obviates the need for labelling. For each species distribution, the code chooses the minimal area that fits all the sites. I find that it reduces the mental overload in comparison to auto-zoom when looking at a bunch of maps in the context of the PDF, but we may swap this for another solution. For now, it’s good enough, and a definite improvement over broken maps.

                                                                                                                                                                                                                              Subscribe
                                                                                                                                                                                                                                -
                                                                                                                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/index.html b/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/index.html index fd0df8a22..8624ed40a 100644 --- a/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/index.html +++ b/til/2023/11/28/our-blue-machine-is-changing-but-we-are-not-helpless/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                Home » TIL: Today I learned...

                                                                                                                                                                                                                                Our Blue Machine is changing, but we are not helpless

                                                                                                                                                                                                                                November 28, 2023

                                                                                                                                                                                                                                Quoting the final chapter of Helen Czerski’s Blue Machine:

                                                                                                                                                                                                                                While doing the research for almost every story in this book, I found that the latest scientific research papers on each topic started by discussing how those systems are changing. The giant ocean engine will keep turning, but its delicate equilibrium and the ways life is woven through it are not fixed. It must turn, but it doesn’t have to turn like this in every detail. And yet this is a system of huge richness, and physical oceanography and evolution would take a long time to replace that bounty if we lost it. We don’t understand everything about the global ocean, but we certainly do understand enough to know how valuable it is and the most obvious ways in which to protect it. The primary reason for setting out the damage we have inflicted on the blue machine is not to shock. It’s to lift us out of helplessness. As we acquire knowledge, we also acquire the grounds for optimism.

                                                                                                                                                                                                                                There’s so much to love about the book, as there is so much to love about the earth’s Blue Machine. The bulk of the book comprises stories that explain how the ocean works – from the physics and chemistry of seawater through to oceanic messengers, passengers, and voyagers. One quote from the book doesn’t do it justice – just go read it yourself.

                                                                                                                                                                                                                                Subscribe
                                                                                                                                                                                                                                  -
                                                                                                                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/index.html b/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/index.html index 60cfd45a3..f091ca85b 100644 --- a/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/index.html +++ b/til/2023/12/14/transfer-learning-applies-to-energy-market-bidding/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                  Home » TIL: Today I learned...

                                                                                                                                                                                                                                  Transfer learning applies to energy market bidding

                                                                                                                                                                                                                                  December 14, 2023

                                                                                                                                                                                                                                  Quoting the abstract of Transferable Energy Storage Bidder by Yousuf Baker, Ningkun Zheng, and Bolun Xu:

                                                                                                                                                                                                                                  Energy storage resources must consider both price uncertainties and their physical operating characteristics when participating in wholesale electricity markets. This is a challenging problem as electricity prices are highly volatile, and energy storage has efficiency losses, power, and energy constraints. This paper presents a novel, versatile, and transferable approach combining model-based optimization with a convolutional long short-term memory network for energy storage to respond to or bid into wholesale electricity markets. We test our proposed approach using historical prices from New York State, showing it achieves state-of-the-art results, achieving between 70% to near 90% profit ratio compared to perfect foresight cases, in both price response and wholesale market bidding setting with various energy storage durations. We also test a transfer learning approach by pre-training the bidding model using New York data and applying it to arbitrage in Queensland, Australia. The result shows transfer learning achieves exceptional arbitrage profitability with as little as three days of local training data, demonstrating its significant advantage over training from scratch in scenarios with very limited data availability.

                                                                                                                                                                                                                                  I’m not sure about the practical implications, but it’s interesting that data from New York is informative for Queensland. I also found the approach of predicting the opportunity value function (rather than forecasting prices) to be clever. However, I’m not familiar with research in the field, so this may be standard practice. Still, I will refer back to this paper and its references if I go deeper into energy market bidding.

                                                                                                                                                                                                                                  Subscribe
                                                                                                                                                                                                                                    -
                                                                                                                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/index.html b/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/index.html index f5fb5d01f..9ae2c4325 100644 --- a/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/index.html +++ b/til/2023/12/18/positioning-is-a-common-problem-for-data-scientists/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                    Home » TIL: Today I learned...

                                                                                                                                                                                                                                    Positioning is a common problem for data scientists

                                                                                                                                                                                                                                    December 18, 2023

                                                                                                                                                                                                                                    I became a data scientist by accident: I followed my curiosity and did a PhD in computational linguistics and recommender systems. When I finished my PhD in 2012, I discovered I could call myself a data scientist rather than a software engineer with a research background (which was a bit of a mouthful). As 2012 was the year Harvard Business Review declared data scientist to be the sexiest job of the 21st century, I didn’t need to think much about what kind of data scientist I was. Just being a data scientist was pretty unique.

                                                                                                                                                                                                                                    The world has changed in the past eleven years, and now there are many more data scientists. While you could earn good money as a generic data scientist, you don’t stand out. That is, it’s not only that software commodities are replacing interesting data science work, and that large language models are making some skills irrelevant – whatever is left of the core data science skillset has become an undifferentiated commodity.

                                                                                                                                                                                                                                    I’ve been thinking a lot about positioning as an independent consultant recently, after realising that the lines between solo consulting and product building are blurry. One great source to learn more on the topic is Jonathan Stark, who has published many valuable resources over the years. Among them, I found a podcast interview he did in May this year with Genevieve Hayes, titled Building Your Authority in Data Science.

                                                                                                                                                                                                                                    Whether you’re an employee or independent data scientist, it’s worth listening to the interview. Here are my key takeaways:

                                                                                                                                                                                                                                    • Even though data scientists are already highly specialised, the problem of positioning oneself and standing out is common.
                                                                                                                                                                                                                                    • Understanding marketing and the business side in addition to mastering the technical skills can be a superpower, as you can act as a bridge between non-technical people and the “nerds”.
                                                                                                                                                                                                                                    • You need to be perceived as meaningfully different by your target audience, regardless of whether you choose to specialise in a horizontal (specific data science skill like computer vision) or in a vertical (specific industry like renewable energy). If your target audience doesn’t find you meaningfully different, you have more work to do.
                                                                                                                                                                                                                                    • Avoid basing your self-worth on where you sit compared to other data scientists. If you’re good enough technically (C to B+) and you have complementary skills and an outcome-driven mindset, you’d be unstoppable. This still seems rare.
                                                                                                                                                                                                                                    • Publish stuff that business people can understand, i.e., connect what you can do on the technical side with business value.
                                                                                                                                                                                                                                    • You don’t need to be managing people to deliver results, e.g., Jonathan chose to remain solo and not hire employees. Focusing on business results is what matters.
                                                                                                                                                                                                                                    • At the time of the interview (May 2023), Jonathan was searching for a ChatGPT consultant to learn whether he could turn his content into a chatbot. He was surprised he could barely find anyone. This is a good example of riding a hype cycle, as being an early authority on ChatGPT can lead to solid business outcomes for indie consultants. However, given the nature of hype cycles, this can change quickly.
                                                                                                                                                                                                                                    • Do things you’re deeply curious about, as enthusiasm helps you stand out. Going down rabbit holes can be a strength. In the context of ChatGPT consulting, this reminded me of Simon Willison and Ethan Mollick, who have become more well-known recently due to their curiosity and blogging on generative AI.
                                                                                                                                                                                                                                    • There are many ways to make money, so you might as well work on something you like.

                                                                                                                                                                                                                                    Personally, I’m still figuring out my positioning. I found it interesting that Genevieve and Jonathan agreed that stereotypical data scientists care more about building their models than about how they’re used. While I enjoy the technical aspects of modelling and other data science tasks, I’m much more interested in shipping work that matters. That’s why I became a data scientist originally – I left academia after my PhD and joined startups to get close to business problems and build stuff people use. If that’s still a rarity, I suppose it can help with my positioning. That said, I’m also exploring a deeper vertical specialisation (currently looking at energy markets).

                                                                                                                                                                                                                                    Subscribe
                                                                                                                                                                                                                                      -
                                                                                                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/01/08/the-power-of-parasocial-relationships/index.html b/til/2024/01/08/the-power-of-parasocial-relationships/index.html index a96c467e9..4dcff0922 100644 --- a/til/2024/01/08/the-power-of-parasocial-relationships/index.html +++ b/til/2024/01/08/the-power-of-parasocial-relationships/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                      Home » TIL: Today I learned...

                                                                                                                                                                                                                                      The power of parasocial relationships

                                                                                                                                                                                                                                      January 8, 2024

                                                                                                                                                                                                                                      I recently learned about the term parasocial relationship from The Business of Authority episode on The “Secret” Benefit of Podcasting. Quoting Wikipedia:

                                                                                                                                                                                                                                      Parasocial interaction (PSI) refers to a kind of psychological relationship experienced by an audience in their mediated encounters with performers in the mass media, particularly on television and on online platforms. Viewers or listeners come to consider media personalities as friends, despite having no or limited interactions with them. PSI is described as an illusory experience, such that media audiences interact with personas (e.g., talk show hosts, celebrities, fictional characters, social media influencers) as if they are engaged in a reciprocal relationship with them. The term was coined by Donald Horton and Richard Wohl in 1956.

                                                                                                                                                                                                                                      A parasocial interaction, an exposure that garners interest in a persona, becomes a parasocial relationship after repeated exposure to the media persona causes the media user to develop illusions of intimacy, friendship, and identification. Positive information learned about the media persona results in increased attraction, and the relationship progresses. Parasocial relationships are enhanced due to trust and self-disclosure provided by the media persona.

                                                                                                                                                                                                                                      Interestingly, I feel exactly that sense of familiarity with Rochelle Moulton and Jonathan Stark, hosts of The Business of Authority. This is despite having no direct interactions with them. I only listened to a few episodes, read some of Jonathan’s materials, and signed up to his daily mailing list (which means he’s now a constant presence in my life).

                                                                                                                                                                                                                                      As both Rochelle and Jonathan have been putting themselves out there for years, sharing quality content, and creating parasocial relationships with their target audience, they can charge premium fees for their expertise. At the time of this writing, Rochelle charges $1000 for a one-hour coaching session, while Jonathan charges $2500.

                                                                                                                                                                                                                                      It’s embarrassingly simple, but incredibly powerful and applicable in a wide range of areas. Or in Jonathan’s words: “It takes time, but if you want to justify premium fees, becoming a celebrity in your space is a great way to grow your profits.”

                                                                                                                                                                                                                                      Subscribe
                                                                                                                                                                                                                                        -
                                                                                                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/index.html b/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/index.html index 5add580cc..825aff8c6 100644 --- a/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/index.html +++ b/til/2024/01/09/psychographic-specialisations-may-work-for-discipline-generalists/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                        Home » TIL: Today I learned...

                                                                                                                                                                                                                                        Psychographic specialisations may work for discipline generalists

                                                                                                                                                                                                                                        January 9, 2024

                                                                                                                                                                                                                                        The Business of Authority is a treasure trove of information for independent consultants. This morning, I listened to a 2019 episode titled Five Ways To Specialize, where hosts Jonathan Stark and Rochelle Moulton went deep into these approaches:1

                                                                                                                                                                                                                                        • Horizontal Specialization: Niching Down on a skill that can be applied to a very broad range of client types. e.g., responsive web design, iOS development, MySQL administration.
                                                                                                                                                                                                                                        • Platform Specialization: A subset of Horizontal Specialization that targets a tool or platform that your ideal buyer was involved in choosing. Shopify, WordPress, or Salesforce are examples of Platform Specializations because your buyer almost certainly was involved in choosing the technology in question. Tools like Photoshop, JavaScript, or MySQL are not Platform Specializations because the buyer probably didn’t directly or consciously choose them.
                                                                                                                                                                                                                                        • Vertical Specialization: Niching Down on a market segment in which vendors offer goods and services specific to an industry, trade, profession, or other group of customers with specialized needs. Typical examples of buyers in a vertical market would be quick service restaurants, ski resorts, pet shelters, auto repair shops, and so on.
                                                                                                                                                                                                                                        • Demographic Specialization: Niching Down on a market segment defined by personal attributes of an individual (e.g., Baby Boomers, New York residents, millionaires, Asian Americans, migraine sufferers).
                                                                                                                                                                                                                                        • Psychographic Specialization: Niching Down on a market segment defined by personal beliefs, attitudes, or behaviors of an individual (e.g., environmentalists, skeptics, flat-earthers, dreamers).

                                                                                                                                                                                                                                        I discovered the episode because I wanted to understand psychographic specialisations better, as part of figuring out my current positioning. In my first round as an independent consultant (back in 2014-2015), I specialised horizontally as a data scientist. These days, data scientists aren’t that special, and I’m not that interested in further horizontal specialisation (e.g., as a machine learning engineer focused on edge applications). While horizontal specialisations are interesting from a technical perspective, my true interest is in nature-positive outcomes, i.e., cater for the psychographic market segment of “environmentalists”.2 This is consistent with career actions I’ve taken over the past decade, e.g., stopping my independent consulting / product building to join Car Next Door (now Uber Carshare) in 2016, founding a sustainability resource group as an Automattic employee in 2020, and leaving Automattic to focus more of my time on climate tech in 2021.

                                                                                                                                                                                                                                        Right now, my homepage says that “I provide independent consulting services around Data & AI, focusing on small-to-medium organisations in the climate tech and nature-positive sector”. However, my website tagline still says “Yanir Seroussi | Engineering Data Science & More” – changed last year from the long-standing “Data Science and Beyond”, but it’s still focused on my discipline rather than on the clients I aim to serve.3 Like many other independent consultants, I find it hard to commit to a niche that feels too narrow!

                                                                                                                                                                                                                                        Back to the podcast episode: To my relief, Stark noted that for those who choose psychographic specialisations, also specialising horizontally is less important. So going broad with Data & AI is both consistent with my experience (I’ve done work all over the stack), and makes sense when it comes to what I care about. For example, I don’t mind doing any web & data work for Reef Life Survey or engaging in full-stack data engineering for Work on Climate, as it delivers nature-positive outcomes in both cases. What I’m missing is higher profitability and a consistent client pipeline, but it seems like fully committing to the psychographic path is the right way to go.

                                                                                                                                                                                                                                        1. The definitions were taken from Stark’s glossary, with slight tweaks. ↩︎

                                                                                                                                                                                                                                        2. In my mind, the word “environmentalist” carries some negative connotations of aggressive and unrealistic activists, which is why I prefer the term “nature-positive” – focusing on the key outcome people in my niche want. ↩︎

                                                                                                                                                                                                                                        3. Stark defines discipline as your craft, specialty, or job title. ↩︎

                                                                                                                                                                                                                                        Subscribe
                                                                                                                                                                                                                                          -
                                                                                                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/02/06/future-software-development-may-require-fewer-humans/index.html b/til/2024/02/06/future-software-development-may-require-fewer-humans/index.html index 925bf3922..add393aa1 100644 --- a/til/2024/02/06/future-software-development-may-require-fewer-humans/index.html +++ b/til/2024/02/06/future-software-development-may-require-fewer-humans/index.html @@ -4,7 +4,7 @@ Well, we’re introducing a new vehicle type. So still a large language model with applications that are built on top of it, but it’s a new vehicle type, and it’s the Ford F-150. And it is specifically built for those environments and those orientations and those jobs.

                                                                                                                                                                                                                                          From a model perspective, there’s a massive, massive difference between us and others in that, one, something that’s tuned for software will know more about software. […] OpenAI has made famous reinforcement learning via human feedback. Anthropic has made famous reinforcement learning via constitutional AI or algorithmic AI and things of that nature. We’re introducing something that we call reinforcement learning via code execution feedback. So taking advantage of the aspects of software and the aspects of code that you might imagine. One, it’s inspectable, two, it’s runnable, and three, you have to compile or execute these things and you can get deterministic feedback. And so what we have done is inside of our training set, we’ve made very, very, very, very different decisions than general purpose models would make.

                                                                                                                                                                                                                                          And this goes to the heart of why a truck is different than a sedan, we’ve made very different design decisions. We have included only high-quality code in the model. We have cut data sources out of the initial dataset because it’s a very different audience that’s going to use this for a very different purpose. And so this goes to the reinforcement learning side of the fence, too. We care very deeply that yeah, we’ll have human feedback as well, but the reinforcement learning is going to be all about what’s produced from the software side. So we’ve taken about 50,000 high quality real-world projects out of the initial training dataset and are using it on the reinforcement side. We’ve made all the git commits executable. We have all three legs of the stool that we need. We’ve got the issue that’s described in real language. We’ve got the code, we’ve got tests, we’ve got all of the things that we need, and we send this through a reinforcement learning platform on our model to see what it does.

                                                                                                                                                                                                                                          I find the idea of reinforcement learning from code execution feedback very compelling. Given how well GPT-4 already performs, it makes sense that you could get much better results by following poolside’s approach.

                                                                                                                                                                                                                                          As with other AI advancements, the implications on society are likely to be profound. It’s not hard to imagine a situation where software development becomes much less labour-intensive than it is today, leading to unemployment among software developers. Or it could be that access to better AI software developers would lead to much more software getting built. Who knows?

                                                                                                                                                                                                                                          My bet is on the future of software development being quite different from today, and on things changing more rapidly than most people realise. Skills that are valuable today may be obsolete within 5-10 years. Rephrasing poolside’s website, we’re likely to switch from human-led-AI-assisted development to AI-led-human-assisted development.

                                                                                                                                                                                                                                          Subscribe
                                                                                                                                                                                                                                            -
                                                                                                                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/index.html b/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/index.html index 0b5eb88d9..228b26efb 100644 --- a/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/index.html +++ b/til/2024/02/17/the-three-cs-of-indie-consulting-confidence-cash-and-connections/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                            Home » TIL: Today I learned...

                                                                                                                                                                                                                                            The three Cs of indie consulting: Confidence, Cash, and Connections

                                                                                                                                                                                                                                            February 17, 2024

                                                                                                                                                                                                                                            Since realising that the lines between solo consulting and product building are blurry, I’ve been consuming a lot of Jonathan Stark’s materials on positioning, value pricing, and related topics. I mostly read and listen to his content, but this short video hits the nail on the head:

                                                                                                                                                                                                                                            The quick summary is that you should have the following before you quit your job to become a solo consultant:

                                                                                                                                                                                                                                            1. Confidence that you can deliver immediate value to your clients.
                                                                                                                                                                                                                                            2. Cash to get you through the tough times.
                                                                                                                                                                                                                                            3. Connections to ideal buyers in your target market.

                                                                                                                                                                                                                                            When I quit my last job, my plan was to take some time off, and then look at building a solo product business. It took me a while to consider consulting as a serious pursuit – I was definitely not thinking of things like ideal buyers in my target market, let alone how to connect to them and maintain relationships over time. I do have the confidence, and I’m not strapped for cash, but getting the positioning right and building connections is a long-term play. Anyone who’s quitting with the intention of consulting a soloist should definitely keep the three Cs in mind!

                                                                                                                                                                                                                                            Subscribe
                                                                                                                                                                                                                                              -
                                                                                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                              Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/index.html b/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/index.html index 2a954b835..f924379da 100644 --- a/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/index.html +++ b/til/2024/03/12/atomic-habits-is-full-of-actionable-advice/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                              Home » TIL: Today I learned...

                                                                                                                                                                                                                                              Atomic Habits is full of actionable advice

                                                                                                                                                                                                                                              March 12, 2024

                                                                                                                                                                                                                                              I recently picked up the Audible version of Atomic Habits, mostly because I kept seeing it everywhere. I wasn’t disappointed. It immediately gave me ideas for improving my routines based on a cursory listen. I will definitely revisit it and refer to resources like the habit cheat sheet in the future.

                                                                                                                                                                                                                                              As a test, I used several tips to form better habits around sending messages (which is often a cause for procrastination):

                                                                                                                                                                                                                                              • Habit stacking: Send one message after my morning workout (chained to an existing positive habit).
                                                                                                                                                                                                                                              • Temptation bundling and reinforcement: After the message is out, have a snack.
                                                                                                                                                                                                                                              • Tracking: Tick a box once the message is sent, which adds up to a lovely streak.

                                                                                                                                                                                                                                              …and it worked! It feels a bit odd consciously training myself like this, but it’s better than the alternative of sticking to bad habits and procrastination.

                                                                                                                                                                                                                                              Subscribe
                                                                                                                                                                                                                                                -
                                                                                                                                                                                                                                                Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/index.html b/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/index.html index 0d0a431cf..a4bad069f 100644 --- a/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/index.html +++ b/til/2024/04/05/the-data-engineering-lifecycle-is-not-going-anywhere/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                Home » TIL: Today I learned...

                                                                                                                                                                                                                                                The data engineering lifecycle is not going anywhere

                                                                                                                                                                                                                                                April 5, 2024

                                                                                                                                                                                                                                                After over a decade of engaging in reluctant data engineering, I’ve come to terms with it being a part of my life for as long as I’m going to be working in Data & AI. As the rise of data engineering happened concurrently with my career in data, I picked up some data engineering skills to solve specific problems. That is, I learned from experience rather than from a formal education program. Recently, I read the book Fundamentals of Data Engineering by Joe Reis and Matt Housley, which has helped consolidate my understanding of the field – including key principles and trends.

                                                                                                                                                                                                                                                Fundamentals of Data Engineering is an essential read for anyone working in data. As it aims to remain relevant for years to come, it doesn’t get into the details of specific tools, which are bound to keep changing. Instead, the book describes the data engineering lifecycle, provides guidelines for data architecture and technology choice, and then goes into details on each of the lifecycle stages.

                                                                                                                                                                                                                                                I will not attempt to summarise the book here, as it is itself a summary of a vast field. However, a couple of key items worth pulling out are the data engineering lifecycle and the principles of good data architecture (which apply to other software systems as well). In a world awash with vendor marketing and AI hype, it’s important to keep these abstractions in mind. In particular, the data engineering lifecycle is unlikely to go anywhere, even if more of the work will be done by AIs (aka automated agents).

                                                                                                                                                                                                                                                Five stages of the data engineering lifecycle:

                                                                                                                                                                                                                                                1. Generation (typically outside the control of data engineers)
                                                                                                                                                                                                                                                2. Storage (underlies stages 3-5 – typically managed by data engineers)
                                                                                                                                                                                                                                                3. Ingestion (typically getting data from external production systems to a central warehouse/lake/lakehouse)
                                                                                                                                                                                                                                                4. Transformation (turning raw ingested data into more useful datasets)
                                                                                                                                                                                                                                                5. Serving (for analytics, machine learning, and reverse ETL – the latter means sending data back to production systems)

                                                                                                                                                                                                                                                Undercurrents of the data engineering lifecycle (which manifest in every stage):

                                                                                                                                                                                                                                                • Security
                                                                                                                                                                                                                                                • Data management
                                                                                                                                                                                                                                                • DataOps
                                                                                                                                                                                                                                                • Data architecture
                                                                                                                                                                                                                                                • Orchestration
                                                                                                                                                                                                                                                • Software engineering

                                                                                                                                                                                                                                                Principles of good data architecture:

                                                                                                                                                                                                                                                1. Choose common components wisely
                                                                                                                                                                                                                                                2. Plan for failure
                                                                                                                                                                                                                                                3. Architect for scalability
                                                                                                                                                                                                                                                4. Architecture is leadership
                                                                                                                                                                                                                                                5. Always be architecting
                                                                                                                                                                                                                                                6. Build loosely coupled systems
                                                                                                                                                                                                                                                7. Make reversible decisions
                                                                                                                                                                                                                                                8. Prioritize security
                                                                                                                                                                                                                                                9. Embrace FinOps
                                                                                                                                                                                                                                                Subscribe
                                                                                                                                                                                                                                                  -
                                                                                                                                                                                                                                                  Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                  Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                  Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                  Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/04/11/linkedin-is-a-teachable-skill/index.html b/til/2024/04/11/linkedin-is-a-teachable-skill/index.html index edc6a3429..06db37ddd 100644 --- a/til/2024/04/11/linkedin-is-a-teachable-skill/index.html +++ b/til/2024/04/11/linkedin-is-a-teachable-skill/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                  Home » TIL: Today I learned...

                                                                                                                                                                                                                                                  LinkedIn is a teachable skill

                                                                                                                                                                                                                                                  April 11, 2024

                                                                                                                                                                                                                                                  As I delve deeper into the world of solopreneurship, I’m learning many valuable lessons about running a solo business. Examples include:

                                                                                                                                                                                                                                                  One person who exemplifies all of the above lessons is Justin Welsh, who is on the path to $10M in revenue through his soloist ventures. I came across Justin via the Fractionals United community, where many members are interested in getting better at LinkedIn.

                                                                                                                                                                                                                                                  Now, it may sound weird to be talking about being “good at LinkedIn”, but I think it describes Justin well – just look at his feed and profile.

                                                                                                                                                                                                                                                  The beauty of being good at LinkedIn is that it’s a skill that’s highly visible. And unlike other people who merely possess LinkedIn skills, Justin has taken the next step to teach it to others. He offers an online course called the LinkedIn Operating System,1 which I just completed.

                                                                                                                                                                                                                                                  Interestingly, despite being somewhat frugal in general, it didn’t take me long to decide to pay for the course. Justin’s visible skills and the course’s testimonials were helpful, and a 30% discount was helpful in getting me over the line (seems like the discounts change constantly). However, I was already a warm prospect because of my realisation that parasocial relationships are important, and because I already post on LinkedIn (I find it much more civil and valuable than other social networks). Therefore, I see getting better at LinkedIn as an investment in my business. Along with posting regularly on my website and mailing list, improving my LinkedIn visibility should help with attracting quality leads.

                                                                                                                                                                                                                                                  I’m still far from implementing all of Justin’s advice (some of which would take months and years of deliberate practice), but here’s a quick outline of the core parts of the course along with my high-level thoughts.

                                                                                                                                                                                                                                                  1. The foundation: Defining your niche, and building an online persona through your backstory, polarisation, and stories.
                                                                                                                                                                                                                                                    • As noted above, having a well-defined niche is key. I’m still working on sharpening mine, but I’m definitely further along than where I was before I fully recognised the need for specialisation.
                                                                                                                                                                                                                                                    • I’m not a fan of online polarisation (one of the reasons I deleted my Twitter account), so I translated polarisation to strong opinions, which is something I definitely have. As a CEO at a startup I worked with once said: “you have the rare qualities of a) very good with data and b) very able to express your opinions – especially when you don’t agree with how something is being done.” In fact, the realisation that as a soloist I don’t have to hide my opinions to fit in with the norms of a single company is quite liberating.
                                                                                                                                                                                                                                                  2. Content creation: Relaying content through leading/discovering/reporting, and systematising content creation and publication.
                                                                                                                                                                                                                                                    • It’s helpful to think of content as falling into the three broad categories of leading, discovering, and reporting. In some areas, I’m comfortable leading based on my experience. In others, I’m happy to discover new things and share them as I go. Reporting is something I’d like to do more of – perhaps in the form of podcast interviews down the track.
                                                                                                                                                                                                                                                    • Again, systems and habits are key to success. Even people who have a natural talent for LinkedIn content creation won’t get anywhere if they don’t post consistently.
                                                                                                                                                                                                                                                  3. Building your tribe: Smart audience interaction and finding relevant people.
                                                                                                                                                                                                                                                    • This part is also about systems that are fairly straightforward to implement. That’s partly why LinkedIn is a teachable skill: Most people don’t bother systematising audience interaction and growth. At its core, this is about human relationships (which I believe will take longer to automate than other skills that will be taken over by AI).
                                                                                                                                                                                                                                                  4. LinkedIn lead capture: Building the profile funnel and hero section, telling prospects what you do, providing social proof, and ending with a call to action.
                                                                                                                                                                                                                                                    • This includes parts that I already knew I wasn’t doing great (like having a generic cover photo). Now I’m more motivated to improve my profile, and have a checklist of items to address. It’s surprising how many people have profiles that can be easily improved – including many accomplished professionals. This highlights how LinkedIn profile building is a skillset that’s orthogonal to one’s technical career skills.
                                                                                                                                                                                                                                                  5. Business workflow: Inbound strategy, outbound strategy, and selling on LinkedIn.
                                                                                                                                                                                                                                                    • Once again, full of helpful tips and actionable advice that I will be implementing. Like much of the better business and marketing advice out there, the emphasis is on building trust and relationships and providing value, rather than on creepy sales tactics. Doing this well takes time and a conscious effort.

                                                                                                                                                                                                                                                  If you’re in a place where you’d benefit from learning how to LinkedIn, definitely check out the course. Even though it’s a few years old at this point, it’s still highly relevant and recommended. I will be revisiting it in the future.

                                                                                                                                                                                                                                                  1. That’s an affiliate link to the course, so I get a cut if you sign up through the link. It’s probably marketing 101 to incentivise people to generate referrals, but I wouldn’t be publishing this post if I didn’t think the course was valuable. ↩︎

                                                                                                                                                                                                                                                  Subscribe
                                                                                                                                                                                                                                                    -
                                                                                                                                                                                                                                                    Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                    Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                    Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                    Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/05/25/adapting-to-the-economy-of-algorithms/index.html b/til/2024/05/25/adapting-to-the-economy-of-algorithms/index.html index 88670587f..d25bb6ab4 100644 --- a/til/2024/05/25/adapting-to-the-economy-of-algorithms/index.html +++ b/til/2024/05/25/adapting-to-the-economy-of-algorithms/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                    Home » TIL: Today I learned...

                                                                                                                                                                                                                                                    Adapting to the economy of algorithms

                                                                                                                                                                                                                                                    May 25, 2024

                                                                                                                                                                                                                                                    I recently read The Economy of Algorithms: AI and the Rise of the Digital Minions, by Marek Kowalkiewicz. It’s a light read that is mostly aimed at business leaders who are looking to adapt to our current age of increasing algorithmic automation (aka AI). However, some of the points are relevant to any human – especially to knowledge workers.

                                                                                                                                                                                                                                                    The main message of the book is that we’re experiencing an economic transition between three economies:

                                                                                                                                                                                                                                                    1. Economy of Corporations: What we had in the 20th century, with corporations being the most powerful entities.
                                                                                                                                                                                                                                                    2. Economy of People: Emerged in the early 21st century, with individuals gaining agency that has enabled us to compete with corporations (e.g., with YouTube influencers becoming more popular than some TV channels).
                                                                                                                                                                                                                                                    3. Economy of Algorithms: Starting in recent years, algorithms have been gaining agency and competing with people and corporations.

                                                                                                                                                                                                                                                    The emergence of a new economy doesn’t immediately obviate the previous economies, as Marek explains:

                                                                                                                                                                                                                                                    In the economy of algorithms, corporations can leverage advanced algorithms to optimise their operations, enhance decision-making and drive innovation, which increases their competitiveness and profitability. Individuals can harness the power of algorithms to develop new skills, create innovative products or services and participate in emerging markets that were once inaccessible to them. And, as you’ll soon find out, algorithms themselves – without active human or corporate control – can buy, sell, collect and invest funds, and perform other activities that were previously reserved for businesses and people.

                                                                                                                                                                                                                                                    The book includes a rich set of examples and stories that illustrate the above points. As everyone is wondering what to do in this period of accelerating change, Marek proposes nine rules for our age. These are grouped under three areas:

                                                                                                                                                                                                                                                    1. Be the minion master: Automate revenue generation
                                                                                                                                                                                                                                                      • Automate relentlessly but mindfully
                                                                                                                                                                                                                                                      • Build an army of digital minions
                                                                                                                                                                                                                                                      • Empower your people
                                                                                                                                                                                                                                                    2. Be relentlessly curious: Evolve continuously
                                                                                                                                                                                                                                                      • Launch new value propositions
                                                                                                                                                                                                                                                      • Start a digital evolution
                                                                                                                                                                                                                                                      • Stay curious
                                                                                                                                                                                                                                                    3. Be boldly optimistic: Saturate relationships with customers
                                                                                                                                                                                                                                                      • Maximise customer value
                                                                                                                                                                                                                                                      • Build digital ecosystems
                                                                                                                                                                                                                                                      • Create a bold future

                                                                                                                                                                                                                                                    Marek calls corporations that do this well RACERS, which stands for Revenue Automation, Continuous Evolution, and Relationship Saturation.

                                                                                                                                                                                                                                                    While some rules only apply to business leaders, a few also apply to humans who are trying to remain economically relevant – especially the rules around embracing automation and evolving continuously. The alternative is to be made redundant. For example, I recently saw a job ad for a copywriter position that required both the ability to produce excellent copy without the aid of AI, and also the ability to produce excellent copy four times faster with AI tools. The same applies – explicitly or implicitly – to any knowledge work that can now be sped up with automation.

                                                                                                                                                                                                                                                    Subscribe
                                                                                                                                                                                                                                                      -
                                                                                                                                                                                                                                                      Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                      Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                      Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                      Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/06/12/the-rules-of-the-passion-economy/index.html b/til/2024/06/12/the-rules-of-the-passion-economy/index.html index d5719002d..896eedd1a 100644 --- a/til/2024/06/12/the-rules-of-the-passion-economy/index.html +++ b/til/2024/06/12/the-rules-of-the-passion-economy/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                      Home » TIL: Today I learned...

                                                                                                                                                                                                                                                      The rules of the passion economy

                                                                                                                                                                                                                                                      June 12, 2024

                                                                                                                                                                                                                                                      I recently read The Passion Economy by Adam Davidson. The book has some good stories, but I felt like it stretched the main idea a bit too much towards the end (especially the part that glorified Google).

                                                                                                                                                                                                                                                      That said, I liked the chapter about the rules of the passion economy, so I’m posting them here for future reference:

                                                                                                                                                                                                                                                      1. Pursue intimacy at scale: Identify what you love and do well, match your passion to those who want it, and listen to customer feedback.
                                                                                                                                                                                                                                                      2. Only create value that can’t be easily copied.
                                                                                                                                                                                                                                                      3. The price you charge should match the value you provide: Price drive costs, value is a conversation, passion pricing is a service, note your best alternative to a negotiated agreement, charge a lot and then earn it, pay may come in other ways than money, keep changing your prices and offerings, salary is a price, the price you charge should feel good to you, pricing low isn’t a strategy, and pricing is your value.
                                                                                                                                                                                                                                                      4. Fewer passionate customers are better than a lot of indifferent ones: Value pricing requires selling to the right people, don’t rush into a niche too quickly, the best customers are those who seek you out (eventually), and passion/pricing/value/customers are all different views of the same thing.
                                                                                                                                                                                                                                                      5. Passion is a story: You’re selling a story – it better be true, always tell the truth, you must tell your story – it is told in every detail of the business.
                                                                                                                                                                                                                                                      6. Technology should always support your business, not drive it: Do what tech and large industry can’t do, tech-driven scale creates space for businesses built on value and passion, and tech tends toward bigness (so stay small).
                                                                                                                                                                                                                                                      7. Know what business you’re in, and it’s probably not what you think: Change your value capture constantly and your value creation slowly.
                                                                                                                                                                                                                                                      8. Never be in the commodity business, even if you sell what other people consider a commodity.

                                                                                                                                                                                                                                                      Much of this aligns with lessons I’ve learned about running a solo business (see references at the top of my post on LinkedIn as a teachable skill). In fact, I learned about the book from an episode of The Business of Authority, a podcast that explored the same ground as the rules across years of inspiring episodes.

                                                                                                                                                                                                                                                      Subscribe
                                                                                                                                                                                                                                                        -
                                                                                                                                                                                                                                                        Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                        Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                        Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                        Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/06/22/dealing-with-endless-data-changes/index.html b/til/2024/06/22/dealing-with-endless-data-changes/index.html index 44504f460..5dd7b1dca 100644 --- a/til/2024/06/22/dealing-with-endless-data-changes/index.html +++ b/til/2024/06/22/dealing-with-endless-data-changes/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                        Home » TIL: Today I learned...

                                                                                                                                                                                                                                                        Dealing with endless data changes

                                                                                                                                                                                                                                                        June 22, 2024

                                                                                                                                                                                                                                                        I recently listened to the Super Data Science podcast episode on MLOps: The Job and The Key Tools (with Demetrios Brinkmann). The full episode is worth a listen, but one part that especially resonated with me is on how MLOps is about dealing with the unpredictable changes created by data. This is a key difference between working with machine learning in production and traditional applications that are less data-intensive.

                                                                                                                                                                                                                                                        Quotes with minor edits:

                                                                                                                                                                                                                                                        And now I think it’s very common for people to understand, okay, you need some kind of change management solutions when you’re playing around with data, when you’re doing things in ML, you need to make sure that if you’re going to put something into production, you test it in every way possible before it goes out there. Otherwise, you are left with that situation that can wind you up in the headlines. And you don’t want that. Nobody wants to be in the head headlines for bad AI uses.

                                                                                                                                                                                                                                                        […]

                                                                                                                                                                                                                                                        I also think that MLOps is a subset of DevOps.

                                                                                                                                                                                                                                                        […]

                                                                                                                                                                                                                                                        It’s easy to get caught up in is thinking that MLOps is just implementing some tools and then you’re good. But really it’s that organizational level and having the reliability, having the ability to put things into production quickly and roll them back if you need to.

                                                                                                                                                                                                                                                        […]

                                                                                                                                                                                                                                                        DevOps has a lot of change management. So when you make changes to code, you have a process and it’s very mature process that goes into that, right? You change code and then you have unit tests and you have integration tests, and you have somebody that’s merging the branch and maybe you look over it and so there’s a human in the loop and then you roll it out slowly and so you can do some kind of feature flags so that this new code goes into production and you make sure that it doesn’t take down the whole website or take down the whole app, whatever it is. Then you have other tools that can monitor if that new feature or that piece of code is actually being used by users.

                                                                                                                                                                                                                                                        None of that exists when it comes to data. Where is all that? So when it comes to data, data’s changing all the time and people are making changes to data all the time, whether it’s way upstream or downstream.

                                                                                                                                                                                                                                                        […]

                                                                                                                                                                                                                                                        Data’s changing continuously, but there’s no integration tests, there’s no unit tests, there’s no type of feature flag on rolling out these data changes. There’s no type of monitoring if the data is actually being used later on or the new data streams, it just can break things. And you see that in a broken ML model that now is making bad predictions or it’s going insanely slow for some reason, or it’s just not hitting the mark. Or you see it in a broken dashboard, you see it at the end product. And so it’s funny to me that, going back to DevOps and that whole idea of change management and having these processes in place so that when you do change something, you can still have the reliability that you are going to be able to push out this change and you don’t have to get a call at 03:00 AM.

                                                                                                                                                                                                                                                        In short, unlike software, data changes in ways you can only manage, rather than fully control. Organisations should recognise this reality and manage data-intensive applications accordingly by adopting MLOps and DataOps practices, in addition to traditional DevOps.

                                                                                                                                                                                                                                                        Subscribe
                                                                                                                                                                                                                                                          -
                                                                                                                                                                                                                                                          Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                          Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                          Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                          Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/index.html b/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/index.html index 6a39c123c..f0aaca074 100644 --- a/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/index.html +++ b/til/2024/06/26/five-team-building-mistakes-according-to-patty-mccord/index.html @@ -2,7 +2,7 @@
                                                                                                                                                                                                                                                          Home » TIL: Today I learned...

                                                                                                                                                                                                                                                          Five team-building mistakes, according to Patty McCord

                                                                                                                                                                                                                                                          June 26, 2024

                                                                                                                                                                                                                                                          I’ve heard about the legendary Patty McCord and her work as Chief Talent Officer at Netflix, but have never looked deeply into it.

                                                                                                                                                                                                                                                          Recently, I listened to an interview with her on The Startup Podcast, titled Five Biggest Team Building Mistakes.

                                                                                                                                                                                                                                                          They all resonated:

                                                                                                                                                                                                                                                          1. Tolerating mediocre performance
                                                                                                                                                                                                                                                          2. Not sharing business context with everybody on the team
                                                                                                                                                                                                                                                          3. Creating a command and control culture
                                                                                                                                                                                                                                                          4. Avoiding difficult conversations
                                                                                                                                                                                                                                                          5. Practising HR theater

                                                                                                                                                                                                                                                          Key quote (emphasis mine):

                                                                                                                                                                                                                                                          I’m supposed to be this big innovator in my field, right? Oh, Patty, she’s the guru of HR. She’s reinvented… [but] I didn’t do anything radical. Here’s what I did that was radical.

                                                                                                                                                                                                                                                          I stopped doing stupid stuff that doesn’t matter. Just stopped.

                                                                                                                                                                                                                                                          Why do we do this thing that everybody on God’s Earth does? Does it make any sense? Does it move the business forward? Does it make a customer happy? Does it increase our profit margins? And does it allow people to do their best work?

                                                                                                                                                                                                                                                          And if I’m asking somebody who I’m paying quarter of a million dollars a year to go ask somebody in finance for permission to spend $5,000 when all the person in finance adds to the equation is, you know, oh, well, that’s $5,012, you can’t do that. They’re gonna spend their brain power figuring out how to work around somebody who doesn’t know what they’re doing in the finance organization rather than solving the problem we need to solve. I mean, it’s just like, what an utter waste of time for everybody.

                                                                                                                                                                                                                                                          If we all stopped doing stupid stuff that doesn’t matter, the world would definitely be a better place.

                                                                                                                                                                                                                                                          Go check out the full episode, it’s gold!

                                                                                                                                                                                                                                                          Subscribe
                                                                                                                                                                                                                                                            -
                                                                                                                                                                                                                                                            Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to +

                                                                                                                                                                                                                                                            Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                            Public comments are closed, but I love hearing from readers. Feel free to contact me with your thoughts.

                                                                                                                                                                                                                                                            Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & diff --git a/til/index.html b/til/index.html index dd343abb4..522d80d8d 100644 --- a/til/index.html +++ b/til/index.html @@ -11,7 +11,7 @@ ">

                                                                                                                                                                                                                                                            TIL: Today I learned...

                                                                                                                                                                                                                                                            Short, rough posts about things I learned, inspired by Simon Willison’s TIL. Subscribe to the mailing list to receive a digest of recent TILs whenever I publish longer-form posts.
                                                                                                                                                                                                                                                              -
                                                                                                                                                                                                                                                              Alternatively, subscribe to RSS feed.

                                                                                                                                                                                                                                                              2024

                                                                                                                                                                                                                                                              June

                                                                                                                                                                                                                                                              Five team-building mistakes, according to Patty McCord

                                                                                                                                                                                                                                                              Takeaways from an interview with Patty McCord on The Startup Podcast.

                                                                                                                                                                                                                                                              June 26, 2024

                                                                                                                                                                                                                                                              Dealing with endless data changes

                                                                                                                                                                                                                                                              Quotes from Demetrios Brinkmann on the relationship between MLOps and DevOps, with MLOps allowing for managing changes that come from data.

                                                                                                                                                                                                                                                              June 22, 2024

                                                                                                                                                                                                                                                              The rules of the passion economy

                                                                                                                                                                                                                                                              Summary of the main messages from the book The Passion Economy by Adam Davidson.

                                                                                                                                                                                                                                                              June 12, 2024

                                                                                                                                                                                                                                                              May

                                                                                                                                                                                                                                                              Adapting to the economy of algorithms

                                                                                                                                                                                                                                                              Overview of the book The Economy of Algorithms by Marek Kowalkiewicz.

                                                                                                                                                                                                                                                              May 25, 2024

                                                                                                                                                                                                                                                              April

                                                                                                                                                                                                                                                              LinkedIn is a teachable skill

                                                                                                                                                                                                                                                              An high-level overview of things I learned from Justin Welsh’s LinkedIn Operating System course.

                                                                                                                                                                                                                                                              April 11, 2024

                                                                                                                                                                                                                                                              The data engineering lifecycle is not going anywhere

                                                                                                                                                                                                                                                              My key takeaways from reading Fundamentals of Data Engineering by Joe Reis and Matt Housley.

                                                                                                                                                                                                                                                              April 5, 2024

                                                                                                                                                                                                                                                              March

                                                                                                                                                                                                                                                              Atomic Habits is full of actionable advice

                                                                                                                                                                                                                                                              I put the book to use after the first listen, and will definitely revisit it in the future to form better habits.

                                                                                                                                                                                                                                                              March 12, 2024

                                                                                                                                                                                                                                                              February

                                                                                                                                                                                                                                                              The three Cs of indie consulting: Confidence, Cash, and Connections

                                                                                                                                                                                                                                                              Jonathan Stark makes a compelling argument why you should have the three Cs before quitting your job to go solo consulting.

                                                                                                                                                                                                                                                              February 17, 2024

                                                                                                                                                                                                                                                              Future software development may require fewer humans

                                                                                                                                                                                                                                                              Reflecting on an interview with Jason Warner, CEO of poolside.

                                                                                                                                                                                                                                                              February 6, 2024

                                                                                                                                                                                                                                                              January

                                                                                                                                                                                                                                                              Psychographic specialisations may work for discipline generalists

                                                                                                                                                                                                                                                              When focusing on a market segment defined by personal beliefs, it’s often fine to position yourself as a generalist in your craft.

                                                                                                                                                                                                                                                              January 9, 2024

                                                                                                                                                                                                                                                              The power of parasocial relationships

                                                                                                                                                                                                                                                              Repeated exposure to media personas creates relationships that help justify premium fees.

                                                                                                                                                                                                                                                              January 8, 2024

                                                                                                                                                                                                                                                              2023

                                                                                                                                                                                                                                                              December

                                                                                                                                                                                                                                                              Positioning is a common problem for data scientists

                                                                                                                                                                                                                                                              With the commodification of data scientists, the problem of positioning has become more common: My takeaways from Genevieve Hayes interviewing Jonathan Stark.

                                                                                                                                                                                                                                                              December 18, 2023

                                                                                                                                                                                                                                                              Transfer learning applies to energy market bidding

                                                                                                                                                                                                                                                              An interesting approach to bidding of energy storage assets, showing that training on New York data is transferable to Queensland.

                                                                                                                                                                                                                                                              December 14, 2023

                                                                                                                                                                                                                                                              November

                                                                                                                                                                                                                                                              Our Blue Machine is changing, but we are not helpless

                                                                                                                                                                                                                                                              One of my many highlights from Helen Czerski’s Blue Machine.

                                                                                                                                                                                                                                                              November 28, 2023

                                                                                                                                                                                                                                                              You don’t need a proprietary API for static maps

                                                                                                                                                                                                                                                              For many use cases, libraries like cartopy are better than the likes of Mapbox and Google Maps.

                                                                                                                                                                                                                                                              November 21, 2023

                                                                                                                                                                                                                                                              October

                                                                                                                                                                                                                                                              Artificial intelligence was a marketing term all along – just call it automation

                                                                                                                                                                                                                                                              Replacing ‘artificial intelligence’ with ‘automation’ is a useful trick for cutting through the hype.

                                                                                                                                                                                                                                                              October 6, 2023

                                                                                                                                                                                                                                                              September

                                                                                                                                                                                                                                                              The lines between solo consulting and product building are blurry

                                                                                                                                                                                                                                                              It turns out that problems like finding a niche and defining the ideal clients are key to any solo business.

                                                                                                                                                                                                                                                              September 25, 2023

                                                                                                                                                                                                                                                              Google’s Rules of Machine Learning still apply in the age of large language models

                                                                                                                                                                                                                                                              Despite the excitement around large language models, building with machine learning remains an engineering problem with established best practices.

                                                                                                                                                                                                                                                              September 21, 2023

                                                                                                                                                                                                                                                              August

                                                                                                                                                                                                                                                              The Minimalist Entrepreneur is too prescriptive for me

                                                                                                                                                                                                                                                              While I found the story of Gumroad interesting, The Minimalist Entrepreneur seems to over-generalise from the founder’s experience.

                                                                                                                                                                                                                                                              August 21, 2023

                                                                                                                                                                                                                                                              Revisiting Start Small, Stay Small in 2023 (Chapter 2)

                                                                                                                                                                                                                                                              A summary of the second chapter of Rob Walling’s Start Small, Stay Small, along with my thoughts & reflections.

                                                                                                                                                                                                                                                              August 17, 2023

                                                                                                                                                                                                                                                              Revisiting Start Small, Stay Small in 2023 (Chapter 1)

                                                                                                                                                                                                                                                              A summary of the first chapter of Rob Walling’s Start Small, Stay Small, along with my thoughts & reflections.

                                                                                                                                                                                                                                                              August 16, 2023

                                                                                                                                                                                                                                                              Email notifications on public GitHub commits

                                                                                                                                                                                                                                                              GitHub publishes an Atom feed, which means you can use any RSS reader to follow commits.

                                                                                                                                                                                                                                                              August 14, 2023

                                                                                                                                                                                                                                                              The rule of thirds can probably be ignored

                                                                                                                                                                                                                                                              Turns out that the rule of thirds for composing visuals may not be that important.

                                                                                                                                                                                                                                                              August 11, 2023

                                                                                                                                                                                                                                                              July

                                                                                                                                                                                                                                                              Using YubiKey for SSH access

                                                                                                                                                                                                                                                              Some pointers for setting up SSH access with YubiKey on Ubuntu 22.04.

                                                                                                                                                                                                                                                              July 23, 2023

                                                                                                                                                                                                                                                              Making a TIL section with Hugo and PaperMod

                                                                                                                                                                                                                                                              How I added a Today I Learned section to my Hugo site with the PaperMod theme.

                                                                                                                                                                                                                                                              July 17, 2023

                                                                                                                                                                                                                                                              You can’t save time

                                                                                                                                                                                                                                                              Time can be spent doing different activities, but it can’t be stored and saved for later.

                                                                                                                                                                                                                                                              July 11, 2023
                                                                                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | +
                                                                                                                                                                                                                                                              Join hundreds of subscribers. No spam or AI-generated slop. Unsubscribe any time.

                                                                                                                                                                                                                                                              2024

                                                                                                                                                                                                                                                              June

                                                                                                                                                                                                                                                              Five team-building mistakes, according to Patty McCord

                                                                                                                                                                                                                                                              Takeaways from an interview with Patty McCord on The Startup Podcast.

                                                                                                                                                                                                                                                              June 26, 2024

                                                                                                                                                                                                                                                              Dealing with endless data changes

                                                                                                                                                                                                                                                              Quotes from Demetrios Brinkmann on the relationship between MLOps and DevOps, with MLOps allowing for managing changes that come from data.

                                                                                                                                                                                                                                                              June 22, 2024

                                                                                                                                                                                                                                                              The rules of the passion economy

                                                                                                                                                                                                                                                              Summary of the main messages from the book The Passion Economy by Adam Davidson.

                                                                                                                                                                                                                                                              June 12, 2024

                                                                                                                                                                                                                                                              May

                                                                                                                                                                                                                                                              Adapting to the economy of algorithms

                                                                                                                                                                                                                                                              Overview of the book The Economy of Algorithms by Marek Kowalkiewicz.

                                                                                                                                                                                                                                                              May 25, 2024

                                                                                                                                                                                                                                                              April

                                                                                                                                                                                                                                                              LinkedIn is a teachable skill

                                                                                                                                                                                                                                                              An high-level overview of things I learned from Justin Welsh’s LinkedIn Operating System course.

                                                                                                                                                                                                                                                              April 11, 2024

                                                                                                                                                                                                                                                              The data engineering lifecycle is not going anywhere

                                                                                                                                                                                                                                                              My key takeaways from reading Fundamentals of Data Engineering by Joe Reis and Matt Housley.

                                                                                                                                                                                                                                                              April 5, 2024

                                                                                                                                                                                                                                                              March

                                                                                                                                                                                                                                                              Atomic Habits is full of actionable advice

                                                                                                                                                                                                                                                              I put the book to use after the first listen, and will definitely revisit it in the future to form better habits.

                                                                                                                                                                                                                                                              March 12, 2024

                                                                                                                                                                                                                                                              February

                                                                                                                                                                                                                                                              The three Cs of indie consulting: Confidence, Cash, and Connections

                                                                                                                                                                                                                                                              Jonathan Stark makes a compelling argument why you should have the three Cs before quitting your job to go solo consulting.

                                                                                                                                                                                                                                                              February 17, 2024

                                                                                                                                                                                                                                                              Future software development may require fewer humans

                                                                                                                                                                                                                                                              Reflecting on an interview with Jason Warner, CEO of poolside.

                                                                                                                                                                                                                                                              February 6, 2024

                                                                                                                                                                                                                                                              January

                                                                                                                                                                                                                                                              Psychographic specialisations may work for discipline generalists

                                                                                                                                                                                                                                                              When focusing on a market segment defined by personal beliefs, it’s often fine to position yourself as a generalist in your craft.

                                                                                                                                                                                                                                                              January 9, 2024

                                                                                                                                                                                                                                                              The power of parasocial relationships

                                                                                                                                                                                                                                                              Repeated exposure to media personas creates relationships that help justify premium fees.

                                                                                                                                                                                                                                                              January 8, 2024

                                                                                                                                                                                                                                                              2023

                                                                                                                                                                                                                                                              December

                                                                                                                                                                                                                                                              Positioning is a common problem for data scientists

                                                                                                                                                                                                                                                              With the commodification of data scientists, the problem of positioning has become more common: My takeaways from Genevieve Hayes interviewing Jonathan Stark.

                                                                                                                                                                                                                                                              December 18, 2023

                                                                                                                                                                                                                                                              Transfer learning applies to energy market bidding

                                                                                                                                                                                                                                                              An interesting approach to bidding of energy storage assets, showing that training on New York data is transferable to Queensland.

                                                                                                                                                                                                                                                              December 14, 2023

                                                                                                                                                                                                                                                              November

                                                                                                                                                                                                                                                              Our Blue Machine is changing, but we are not helpless

                                                                                                                                                                                                                                                              One of my many highlights from Helen Czerski’s Blue Machine.

                                                                                                                                                                                                                                                              November 28, 2023

                                                                                                                                                                                                                                                              You don’t need a proprietary API for static maps

                                                                                                                                                                                                                                                              For many use cases, libraries like cartopy are better than the likes of Mapbox and Google Maps.

                                                                                                                                                                                                                                                              November 21, 2023

                                                                                                                                                                                                                                                              October

                                                                                                                                                                                                                                                              Artificial intelligence was a marketing term all along – just call it automation

                                                                                                                                                                                                                                                              Replacing ‘artificial intelligence’ with ‘automation’ is a useful trick for cutting through the hype.

                                                                                                                                                                                                                                                              October 6, 2023

                                                                                                                                                                                                                                                              September

                                                                                                                                                                                                                                                              The lines between solo consulting and product building are blurry

                                                                                                                                                                                                                                                              It turns out that problems like finding a niche and defining the ideal clients are key to any solo business.

                                                                                                                                                                                                                                                              September 25, 2023

                                                                                                                                                                                                                                                              Google’s Rules of Machine Learning still apply in the age of large language models

                                                                                                                                                                                                                                                              Despite the excitement around large language models, building with machine learning remains an engineering problem with established best practices.

                                                                                                                                                                                                                                                              September 21, 2023

                                                                                                                                                                                                                                                              August

                                                                                                                                                                                                                                                              The Minimalist Entrepreneur is too prescriptive for me

                                                                                                                                                                                                                                                              While I found the story of Gumroad interesting, The Minimalist Entrepreneur seems to over-generalise from the founder’s experience.

                                                                                                                                                                                                                                                              August 21, 2023

                                                                                                                                                                                                                                                              Revisiting Start Small, Stay Small in 2023 (Chapter 2)

                                                                                                                                                                                                                                                              A summary of the second chapter of Rob Walling’s Start Small, Stay Small, along with my thoughts & reflections.

                                                                                                                                                                                                                                                              August 17, 2023

                                                                                                                                                                                                                                                              Revisiting Start Small, Stay Small in 2023 (Chapter 1)

                                                                                                                                                                                                                                                              A summary of the first chapter of Rob Walling’s Start Small, Stay Small, along with my thoughts & reflections.

                                                                                                                                                                                                                                                              August 16, 2023

                                                                                                                                                                                                                                                              Email notifications on public GitHub commits

                                                                                                                                                                                                                                                              GitHub publishes an Atom feed, which means you can use any RSS reader to follow commits.

                                                                                                                                                                                                                                                              August 14, 2023

                                                                                                                                                                                                                                                              The rule of thirds can probably be ignored

                                                                                                                                                                                                                                                              Turns out that the rule of thirds for composing visuals may not be that important.

                                                                                                                                                                                                                                                              August 11, 2023

                                                                                                                                                                                                                                                              July

                                                                                                                                                                                                                                                              Using YubiKey for SSH access

                                                                                                                                                                                                                                                              Some pointers for setting up SSH access with YubiKey on Ubuntu 22.04.

                                                                                                                                                                                                                                                              July 23, 2023

                                                                                                                                                                                                                                                              Making a TIL section with Hugo and PaperMod

                                                                                                                                                                                                                                                              How I added a Today I Learned section to my Hugo site with the PaperMod theme.

                                                                                                                                                                                                                                                              July 17, 2023

                                                                                                                                                                                                                                                              You can’t save time

                                                                                                                                                                                                                                                              Time can be spent doing different activities, but it can’t be stored and saved for later.

                                                                                                                                                                                                                                                              July 11, 2023
                                                                                                                                                                                                                                                              Text and figures licensed under CC BY-NC-ND 4.0 by Yanir Seroussi, except where noted otherwise  | Powered by Hugo & PaperMod
                                                                                                                                                                                                                                                              \ No newline at end of file