Integrating Machine Learning into products – Insights from traditional algorithm-based products

Introducing Machine Learning (ML) into products presents unique challenges for product managers. This article explores some of the parallels between ML-based products and traditional algorithm-based products in the mobile telecommunications sector. It describes some of the product management challenges and presents strategies successfully employed in managing algorithm-based products that can be adapted for ML integration.

Existing non-ML, algorithm based products

Algorithm-based software products have been prevalent across many sectors for decades. In this article, we focus on such products in the Mobile telecommunications sector. In the mobile telecommunications, software products utilise algorithms to model wireless communication, enabling users to assess the state of the wireless network under varying configurations and mobile traffic conditions. Given the dynamic and unpredictable nature of wireless channels and network traffic, statistical approaches such as Monte Carlo simulations are employed to determine the most probable outcomes amidst uncertainty.

Moreover, in mobile telecommunications, the continuous evolution of technology standards adds another layer of complexity. Each new version of the standard introduces updates to existing technology features or introduces new ones. Product managers often have to respond swiftly to these evolving standards, sometimes relying on partial information.

Introducing ML in products

ML has the potential to enhance various aspects of existing products by introducing new capabilities and improving usability, accuracy, and performance. However, ML also brings complexities and uncertainties, primarily due to two key factors:

1. Dependency on data and partial information

2. Unpredictable nature of outcomes

While the use of large data volumes is more specific to ML, the unpredictability of outcomes is a familiar challenge in products leveraging probabilistic models.

Similarities in challenges: ML based and non-ML, algorithm based products  

Based on my experience, two key similarities exist between ML-based and traditional algorithm-based products:

1. Uncertainty of Outcomes: In ML-based products, this uncertainty arises primarily from the input data. In algorithm-based products referred to in this article, it stems from statistical modeling techniques like Monte Carlo simulations.

2. Availability of Partial Information: For ML-based products, this challenge is due to the quality of data. In algorithm-based products discussed in this article, it arises from rapidly evolving technology standards and partial implementation details.

Strategies for addressing the challenges                    

Below are some of the strategies that have been successfully employed by us to overcome the challenges discussed:

I will elaborate on these with examples:

In the mobile telecoms sector, the introduction of 5G has been a key event in recent years. For software vendors in the sector, this has meant ensuring that their products align with the latest technology features being continuously introduced in the 5G standards.

As with a lot of products, the customers (who in this case are the network operators) expect the latest and greatest capability to be available in the software product as part of the RFx processes. As vendors, it is our responsibility to ensure that the new capability is introduced in a timely manner and is accurate as well as performant.

Over the past 5 years, we have managed to ensure that our product has been the most 5G ready product in its market. We have also introduced a few first-to-market features. This has been achieved by adopting the strategies outlined above:

1. Phased introduction of feature/capabilities across multiple product releases:

In order to minimise the risk of incorrect implementation in the presence of rapid evolution and (at times)  partial information of standards, it is important to divide the capability into phases and focus on the minimum core capability for the initial release of the feature.  This helps ensure that the foundational capability is perfected before expanding into additional and ‘nice-to-have’ capabilities. It also optimises the use of R&D resource and the introduction of code in the code base.

For instance, when introducing new 5G technology features, we used a combination of the following approaches:

·        Introduced capabilities through scripts before adding them to the user interface.

·        Introducing configuration data through files instead of adding these items to the database and the UI.

·        Explored if ‘workable solutions’ or ‘workarounds’ could be possible through minor adjustments to existing capabilities

·        Limiting the outputs to the core outputs and placed further ‘nice-to-have’ outputs on the roadmap for future releases.

1. Maturity labels:

Clear communication and transparency are vital when introducing new capabilities, especially complex features. Assigning maturity labels to new features has proven effective. Commonly used labels include:

·        Proof of Concept (PoC): The feature can be demonstrated to potential users.

·        Field Trial (FT): The core capability is available for validation with a select subset of customers.

·        General Availability (GA): The feature has been validated with a select subset of users, and issues identified during this phase have been resolved.

We typically release new features with an FT label for validation with a select group of customers.

1. Field trials with a subset of friendly customers:

Complex features, particularly those based on statistical processes or partial implementation details, should be validated with a select group of customers through a Field trial before making it generally available. Field trials help validate the accuracy of implementation and provide insights into usability and performance considerations. The number of customers, their selection, and the duration of the trial should vary based on the complexity of the feature. One of the biggest challenges of the field trial phase is not the identification of the customer but ensuring sufficient commitment from the customer.

Some of the approaches that have helped us execute field trials efficiently include:

-            A clear plan and definition of roles and responsibilities at the start.  

-            Clear mechanism for providing feedback.

-            A single platform for all stakeholders to have access to the trial environment

-            Defined processes for updating code to be available on the trial environment

In our case, we validated features with 1 to 2 customers before transitioning to GA status. This approach balances internal resource consumption and time-to-GA.

The trial phase allows us to compare our product's outputs with those from other sources, such as real networks or alternative products, and make necessary adjustments. Overall, the FT allowed us to modify our calculations, understand the workflow and adjust the user interface and gain insights into the run time with real world data.

In some cases, these trials provided insights into new usecases as a result of working with the customer. Once the trial was completed, the feature was transitioned to GA status in the subsequent release.

1. Rapidly addressing issues during the field trial phase:

During the field trial phase, mechanisms and processes should exist to rapidly fix major and critical issues and make the changes available on the trial system.

In conclusion

As we explore the potential of integrating ML into existing products and navigate the unique challenges associated with ML-based features, we can leverage proven strategies from algorithm-based product management to support efficient and effective product development.