Skip to content

ROCKFlows/Rockflows-Overview

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
docs
docs
 
 
README.md
README.md
 
 

Repository files navigation

Rockflows-Overview

Welcome to the Rockflows organization on GitHub! This repository serves as the central hub for all our projects, providing an overview and links to each individual repository.

About Rockflows

Rockflows aims to help non-expert users on the selection of the Machine Learning Workflows to solve their Machine Learning problems.

We pursue several operational objectives:

  1. Constructing a suitable workflow based on data and business requirements.
  2. Retrieving similar past experiments based on data information, business requirements, and algorithmic choices.
  3. Automatically enriching the knowledge base from past experiences.
  4. Discovering new knowledge by exploring past experiments.
  5. (In progress) Automatically determining the profile of an ML workflow.
  6. (In progress) Automatically verifying that an ML workflow adheres to best practices.
  7. (In progress) Automatically verifying that an ML workflow meets FATES properties (during construction or deployment).
  8. (In progress) Automatically justifying best practices through proof construction and justification diagrams.

The project has been ongoing for nine years, exploring various dimensions evident in the different repositories listed below. Today, we aim to provide a more structured framework, leveraging the lessons learned.

Below is a list of our main repositories, each accompanied by a brief description and an abstract of lessons learned.

Repository List, History and Lessons Learned

Below is a list of our main repositories, each accompanied by a brief description and an abstract of lessons learned.

  • 2016 - Comparison of Workflows: a Step Further
    The questions focus on predicting key performance metrics (accuracy, execution time, and memory usage) of machine learning workflows on user datasets without undergoing the evaluation phase.

    • Models relations entre algo, XP et ranking
    • Lessons Learned:
      • Comparing workflows is not about testing if one metric has a better value, whether once or on average, but about assessing if there is a significant difference. Consequently, it is more interesting to build classes of algorithms for a given problem and metric, which requires storing a large number of results and computations.
      • To evaluate a workflow, it is essential to master the evaluation processes, notably by reproducing not only the experiment but also by conducting it on the same data split.
      • Preprocessing has a significant impact; it is not about comparing algorithms but comparing workflows.
      • Metadata dependencies seem difficult/impossible to identify through logical patterns.
    • Publications:

  • 2016, 2022 - Automatic Feature Model Extraction in Machine Learning Libraries
    Since the project's inception, we have aimed to use automated methods to extract Feature Models (FM) from Machine Learning (ML) libraries. To date, we have yet to find a satisfactory approach.
    Our initial attempt in 2016 involved a simple process to extract the structure of the Weka library and convert it into a Feature Model for variability analysis. This approach relied on Java reflection to gather information: (i) Utilizing package organization to determine the library's structure. (ii) Exploiting the "capabilities" of each class to identify algorithm features. (iii) Merging extracted FMs from each class using Familiar.
    This bottom-up approach aimed to merge class-level FMs into a global FM. The complexity of merging FMs exploded, rendering the approach non-viable.
    Attempts to replicate this approach on other ML libraries yielded unsatisfactory results. Further efforts involved extracting algorithm preconditions from RapidMiner and associating them at the hierarchical level to simplify FMs. We used a similar approach for post-conditions by studying algorithm chaining. This led to identifying some pre- and post-conditions but highlighted significant challenges.

    • Models peut etre en faisant une synthese algo-> feature et hierarchie ? Mais justement ce n'est pas aussi simple... on pourrait revenir sur Algo et dessous les implementations? + les relations aux méta-données?
    • Lessons Learned:
      • Bottom-Up Approach Limitations The bottom-up approach is not optimal for extracting FMs from ML libraries because each library's unique structures make the process highly complex.
      • The Need for Comprehensive Reverse Engineering A deep reverse engineering effort is crucial for extracting FMs from ML libraries, with clear objectives guiding the process. An FM's structure serves a specific purpose, distinct from merely locating an algorithm within a library. Additionally, the richness of an FM lies in its structure and constraints, which are particularly challenging to extract automatically.
      • Complexity of Merging Feature Models This issue underscores the broader problem of modeling, where automated merging of diverse structures is highly complex and often impractical.
    • sub-conclusion: Reflecting on these lessons, we recognize the necessity of refining our approaches and considering more top-down or hybrid strategies for future attempts at FM extraction from ML libraries.
    • Publications:

  • 2017 - Learning from experiments on Machine Learning Workflows
    As we cannot predict performances from a logic approach, we deviate to the construction of a Meta-Learning system. The theoretic model we have designed is focused on supervised classification problems, although it can be extended to unsupervised or supervised regression problems. We used the results of different Machine Learning experiments, precisely the results of 1086 workflows (composed of 16 pre-processing techniques, 68 classifiers, and 10 parameter strategies) tested over 101 datasets from the UCI repository. This input data was processed to produce the meta-datasets, with which we trained different sub-models to predict three properties: accuracy, total time, and model size.

    • Models reprendre exactement la spec de Miguel
    • Lessons Learned:
      • Meta-Feature selection and extraction: Based on the learning phases, we developed a module to extract the 48 most representative meta-features of a given dataset, which influence accuracy. It should be noted that the most significant meta-features vary depending on the workflow. We used these meta-features for prediction by leveraging the similarities between the meta-features of the given dataset and those of datasets already in the Meta-Learning system. Calculating meta-features is not free, and selecting the most impactful meta-features is necessary to reduce the computational load.
      • Scalability of the Meta-Learning Model: We focused on designing a highly scalable Meta-Learning model that does not require additional computations before adding a new instance to the knowledge base. So, we enhanced scalability by using regression algorithms instead of classification ones and rejected online learning as we found no incremental learning algorithm for regression in the literature that performs as well as its batch learning counterpart. However, after multiple experiments, we used one sub-model L_meta i,j for each classifier i and property j. For scalability purposes, we included preprocessing techniques and other workflow steps (e.g., parameter strategy) in the meta-features.
    • Sub-conclusion: Recognizing the True Scale of Computational Demands in Research
      • Beyond the limitations related to the number of datasets used and the predefined workflows, it became evident that, in line with autoML research, the required learning processes, i.e., the experiments needed to achieve our goals, exceeded the reasonable capacities of the computing grids at our disposal.
      • Ultimately, (i) we were not truly learning by merely predicting performance without gaining real insights, and (ii) to avoid learning on new datasets, we needed to train on a large number of datasets.
    • Publications:

  • 2016-17 - Towards a software product line for machine learning workflows

  • 2018 -Automating the Learning Processes for Extensive Experimentation
    Given the large number of experiments required, it is essential to automate the learning processes. This automation includes composing pipelines, training on appropriate datasets, and recording measurements. Since it is not sufficient to compare a single set of measurements but rather to compare sets of measurements, it is also necessary to automate ranking and meta-learning processes. To ensure precision, we must maintain consistency in execution and measurement environments. Additionally, considering our limited resources, we aim to reuse data preparation and implement a form of crowd experimenting to add new experiments to our knowledge base at a constrained cost.

    • Models soit l'architecture soit le graphe des algos
    • Lessons Learned:
      • Automating valid Pipeline Composition based on Constraints and Graph Representation enhances the automation of learning processes, ensuring valid ML workflow compositions while efficiently managing complexity.
        • Data-centric constraints are expressed as pre- and post-conditions on algorithms relative to metadata. For example, a pipeline like "NumericTransformationToNominal; NumericNormalization; SVM" is invalid because the preprocessing step NumericNormalization requires numeric data, which the previous algorithm removes.
        • Pipeline-Centric Constraints define a partial order between algorithms. For instance, the constraint "Missing value removal must always precede normalization" invalidates the pipeline "Normalization; ...; MissingValueRemoval; SVM." These constraints significantly reduce the combinatorial complexity between datasets and pipelines.
        • A Graph structure is used to handle the mass of compositions, incremental addition of new algorithms, and strategic graph traversal for launching experiments. We were unable to design it using constraint solvers, as they never scaled effectively. Ensuring Consistent and Scalable Experimentation with Docker and Efficient Resource Management
      • Leveraging Docker Encapsulation to Manage Reproducibility and Resource Limitation:
        • Docker is used to encapsulate the execution environment, ensuring reproducibility and resource limitation. Each algorithm is encapsulated in a Docker image, and the pipeline is executed in a container. This approach ensures that the pipeline is executed in a controlled environment, with limited resources, and that the results are reproducible.
        • Aggregating Metrics is crucial and requires significant modeling effort.
        • Tasks are considered as idempotent, ensuring that repeated executions produce the same outcome. Intermediate results are saved to prevent redundant task executions within workflows. The scheduler handles task dependencies to maintain the correct execution order of workflows, thus avoiding unnecessary re-executions. Users only need to declare the pipelines they wish to execute, with dependency management handled seamlessly in the background.
      • Embracing Errors and Monitoring for Enhanced Learning Detecting and tracking errors among numerous results is pivotal for extracting valuable insights, such as pinpointing missing algorithm preconditions. Traditional monitoring tools have proven invaluable: Integrated Prometheus offers real-time monitoring to uphold scheduler functionality. Grafana provides graphical insights into execution processes, enhancing monitoring capabilities.
        • These safeguards are essential for developers to remain aware of issues and avoid insecure situations. Treating error management as a functional concern within the study of extensive experimentation should be a prerequisite for instilling confidence in the system and viewed as an opportunity for enrichment.
      • Event-Driven Scheduler for Automated Experimentation An event-driven scheduler orchestrates automated pipeline compositions based on new algorithms or datasets, deciding the optimal order to minimize evaluations.
      • Empowering Experimentation with DSL for Pipeline Description The implementation of a domain-specific language (DSL) has empowered users by simplifying pipeline descriptions and enhancing readability, mastery, and control. This DSL facilitates efficient pipeline articulation and generation, ensuring independence from the target language while promoting operational clarity and control.
        • It was a great tool for Rockflows, but we struggled to promote it, especially encountering resistance from data scientists' established practices.
      • Integrating External Experiments requires Rigorous Reproducibility Evaluation We enriched our training dataset by integrating results from our legacy experimentation system into a new metrics registry. This process involved rigorous Reproducibility evaluation to ensure that the added elements were comparable. Additionally, experiments from third parties, like OpenML100, were included after verifying their compatibility with our processes.
    • sub-conclusion: Balancing Research Ambition with Engineering Feasibility in Academic Contexts
      • The automation of learning processes is essential for extensive experimentation. It is crucial to ensure valid pipeline compositions, manage resources efficiently, and maintain reproducibility. Docker encapsulation, error monitoring, and event-driven scheduling are key components of this automation. A domain-specific language (DSL) can simplify pipeline descriptions, but promoting it may be challenging. Integrating external experiments requires rigorous reproducibility evaluation.
      • Despite all this automation work, at this stage of the project, we have a large code base spread across many repositories, deployed on different virtual machines, and accessible externally. However, we lack the means for proper maintenance or to transition to widespread adoption. This effort serves as a springboard for discussions with industry partners but needs to be more utilized in the development foundation. Today, we understand that we risk losing control over the system, which was not apparent to us at the time, explaining subsequent challenges.
      • Le domaine évolue de plus en plus vite, il faut trouver un moyen de capturer la connaissance de manière plus "automatique" mais sans basculer dans le meta-learning.
      • On est conscient que le projet devient gros on tente une exploration des dépendances
    • Publications:

  • Interface web un element clef rapport de Joel -- RF V1 + travail de simon + eddy + .... un composant réutilisable...

    • sub-conclusion:
      • On ne peut pas gérer les changements de verions d'outils tels que angular !

  • Version Mireille : impliqué les data-scientists; Au coueur des feature Models

  • Lagardere

  • Version Nicolas L.

  • 2020 - Deriving New Insights from ML Workflow Modeling While we had established the basics of generating valid ML pipelines by configuring a feature model in a predefined order, we embarked on a journey to facilitate pipeline construction for data scientists using a novel approach to configuration. This approach not only aimed to make the process more intuitive but also to provide new insights through more user-friendly tools, such as pipeline modeling. Despite having a dedicated language at this stage, we chose to explore the path of graphical modeling, aligning with the BPMN standard, to bring a new perspective. Beyond code generation, we faced two challenges: (i) extending the BPMN language to allow the input of additional, coherent information helpful in enriching our knowledge base (e.g., new 'qualified' algorithms and constraints on orders) ; (ii) automatically updating our knowledge base with new information and managing potential conflicts.

    • Models BPMN mais faire le lien avec la hierarchie...
    • Lessons Learned:
      • Custom Merging Approach Required A classical merge approach is unsuitable as individuals can perceive each algorithm differently. We utilized our access to the Feature Model (FM) structure to implement an ad-hoc algorithm, which still needs to be fully validated.
      • Dual Formalization for Algorithm Constraints Understanding the complexities involved, we adopted a dual formalization for algorithm constraints. This thorough approach, driven by the algorithms and the organization of the pipelines, was crucial in managing the intricacies of the project.
    • sub-conclusion: Challenges of Managing Growing Codebases We are increasingly dealing with large volumes of code, and despite significant efforts in structuring, we are beginning to see projects proliferate. This project was initiated from scratch by a second-year undergraduate student for pedagogical purposes and is likely to be repeated, leading to our results becoming increasingly scattered.
    • Publications:

  • 2021-2024 - Evolvable SPL management

  • 2023 - (In progress) Taming the Diversity of Computational NotebooksIn response to the practices of data scientists, who typically prefer starting from notebook analysis and acknowledging the industrial challenge of finding suitable notebooks that meet business requirements (beyond just data), we focused on retrieving "adapted" notebooks based on the expression of business requirements and automatic analysis of datasets.

  • 2024 -(In progress) vers MLOPS et LLMs

  • Version Yacine

About

Welcome to the Rockflows organization on GitHub! This repository serves as the central hub for all our projects, providing an overview and links to each individual repository.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

4 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors