Could you tell us a little about what BMC does and your role within the company?
BMC delivers industry-leading software solutions for IT automation, orchestration, operations, and service management to help organisations free up time and space as they continue to drive their digital initiatives forward. We work with thousands of customers and partners around the world, including more than 85% of the Forbes Global 50.
I’m part of the Solutions Marketing organisation within our Digital Business Automation Business Unit. My primary area of focus is leading our data and cloud market strategy for our Application and Data Workflow Orchestration products, Control-M and BMC Helix Control-M.
What have been the latest developments at BMC with regards to data and multi-cloud?
If you’ve just been casually glancing through your newsfeed over the last couple of years, then you’ve surely seen headlines on how companies are investing heavily in modern data technologies particularly around AI and ML. What has not been as prominent in the news is the fact that, despite the level of investment, focus and sponsorship of the C-Suite, the report card for these initiatives is less than stellar.
A recent stat from a Gartner study on modern data projects shows that only 15% of use cases leveraging AI techniques (such as ML and DNNs) and involving edge and IoT environments will be successful. In other words, the failure rate in such initiatives is about 85%. In another survey run by McKinsey of major advanced analytics programs, they found that 80% of companies’ time in analytics projects is spent on repetitive tasks such as preparing data, whereas the actual value-added work is limited. Moreover, just 10% of companies believe they have this issue under control.
These results are eye popping and immediately call into question why the failure rate is so high despite the investment and focus. There are many contributing factors to this, including the complexity of these projects, the global shortage of data scientists, cloud architects and data engineers required for such initiatives, and – the one that stands out the most to us at BMC – that many of these projects are failing because companies are struggling to operationalise these projects at scale in production.
The industry response to the operationalisation problem has been to develop modern Ops models. This has led to the emergence of DataOps, MLOps etc. DataOps is the application of Agile Engineering and DevOps best practices to the field of Data Management, to rapidly turn new insights into fully operationalised production deliverables that unlock business value from Data.
Within DataOps an important discipline is Orchestration, or in other words the ability to run a complex set of interdependent steps in a data pipeline across four major phases of any data project, Data Ingestion, Data Storage/Processing, and Data Analytics. Control-M from BMC has been a market leader in the application workflow orchestration space for a long time and recently we have invested heavily in making sure that Control-M can continue to serve as the layer of abstraction for orchestrating complex data pipelines from a single point of control so that data initiatives can be operationalised at scale in production. It is important to note that without achieving scale in production no project will be able to deliver the intended value.
What have been the biggest trends you’ve seen developing in automation?
Companies are investing in automation to drive speed and acceleration of business outcomes. Many industries are seeking speed and agility to adapt to a very challenging competitive landscape. For example, Jamie Dimon, the CEO of JP Morgan Chase wrote, in the annual letter to the shareholders in 2020 that banks are competing against a large and powerful shadow banking system. And they are facing extensive competition from Silicon Valley, both in the form of FinTech and Big Tech. One of their strategies to stay competitive is to invest heavily in AI/ML and subsequently in automation not just in the Data and Analytics space but all aspects of their business. It was interesting to see in Dimon’s letter to the shareholders that an entire section was titled “AI, the cloud and digital are transforming how we do business.”
It just goes to show that technology is not just the job and purview of CIOs and CTOs, but the CEOs consider it part of their business strategy. So, the main thing for us to understand as a global supplier of automation and orchestration technology is that our customers are investing in technology to drive business outcomes faster and at scale and not just to gain back-office efficiencies. As a result of all of this we see a high degree of importance being put on aligning technology investments to tangible business outcomes.
At a technological level we see massive adoption of public cloud but at the same time most companies will continue to run many critical systems on-premises and even if the plan is to go all-in on cloud it will be a multi-year journey. This means that a successful orchestration and automation strategy will need platforms that can orchestrate across a hybrid and highly heterogeneous environment that is likely to be multi-cloud but will also include on-premises applications including applications running on the Mainframe in certain industries.
What are the main challenges companies face when starting to build scalable automation strategies? And how can these be overcome?
A big challenge that companies are facing with the rapid adoption of public cloud and many open-source projects is that they have massive technology sprawl, we often describe it as an ever-growing spaghetti bowl of tools. This is primarily because as companies adopt modern technology, they are not necessarily retiring what they adopted years ago. For example, we often see that new systems of engagement are being developed with a modern tech stack, but transactional systems and systems of records such as ERPs and core business applications are still there as they are critical to running the business. This dynamic can often lead to silos of automation because the teams working on the modern, innovative technologies are frequently separated, often by design, from the teams that run what you might consider core business applications.
For expedience, many of the teams working on the modern tech stack will choose the automation tools that they are most familiar with and are used to leveraging. Ultimately this results in a scenario where they have many tools for automation but none of them can automate and orchestrate across the underlying architecture of disparate applications to deliver the intended business outcome. When something goes wrong in production it is excruciatingly difficult to find out where the problem lies as you don’t have a cockpit style view into the workflows that automated the business outcome.
Addressing this problem has been a cornerstone of our strategy with Control-M. We aim to provide customers the ability to automate and orchestrate critical workflows across highly heterogenous environments so that business outcomes are not only automated but as they on-board new business services powered by modern technologies, they don’t have to rewire their automation and orchestration strategy every time.
Many companies understand the value of having a single pane of glass for orchestration but find it difficult to put it in practice because automation and orchestration patterns were not considered during the design phase of their engineering cycle. It is imperative to consider how business outcomes will be operationalised in production at a very early stage in the engineering lifecycle, in other words take a strategic approach to running production.
You’ve advised numerous companies on how to build scalable automation strategies for cloud and data initiatives. Are there any examples of companies that you think have done a particularly good job with this?
I often quote the case study we did with The Hershey Company as a great example of driving business outcomes by focusing on orchestration and automation as a strategic initiative. Many years ago, they standardised on Control-M as their orchestration and automation platform to manage the digital interactions that are necessary to run their business – not just manufacturing, supply planning, supply chain, warehousing, and distribution but also finance, payroll, costing, human resources, marketing, and sales. This allows them to achieve not only scale in production but manage the interdependencies of workflows across all these business functions.
As one of the largest chocolate manufacturers in the world they run a highly complex, data-driven supply chain function, involving daily decisions on production quantities and scheduling shipments to warehouses and distribution centers. Any disruption in these processes can have a significant negative impact on the business, such as delays in shipments and unstocked shelves in sales outlets. To prevent such issues, Hershey relies on Control-M, as a centralised workflow orchestration and automation platform. Control-M not only runs complex interdependent workflows based on events and time-based schedules but also allows them to detect and address problems before they affect the business.
A few years ago, they wrote a blog on their orchestration strategy and in the blog, they shared that one-day Control-M could not trigger workflows on one of their largest SAP instances as SAP became hung in the middle of the day. The action halted the entire SAP landscape for that instance. The application owners might not have become aware of the problem for hours. Fortunately, Control-M detected the issue and alerted the appropriate people. They addressed the problem within minutes and Control-M could continue running the downstream steps in the supply chain workflow, averting potential impact to their business.
What advice would you give to companies that are trying to align data with specific business goals and make better use of insights?
As I mentioned earlier, with the emergence of DataOps and MLOps, the industry is clearly recognising that a strong focus on Ops will be a crucial ingredient in the recipe for success of modern AI and ML projects. It is vital to take orchestration patterns in production as a practice within data engineering, otherwise putting scalable orchestration and operational models at the end of the engineering project will be extremely difficult to retrofit.
What developments from BMC can we expect to see in the year ahead?
Our strategy for Control-M at BMC will stay focused on a couple of basic principles:
- Continue to allow our customers to use Control-M as a single point of control for orchestration as they onboard modern technologies, particularly on the public cloud. This means we will continue to provide new integrations to all major public cloud providers to ensure they can use Control-M to orchestrate workflows across three major cloud infrastructure models of IaaS, Containers and PaaS (Serverless Cloud Services). We plan to continue our strong focus on serverless, and you will see more out-of-the-box integrations from Control-M to support the PaaS model.
- We recognise that enterprise orchestration is a team sport, which involves coordination across engineering, operations and business users. And, with this in mind, we plan to bring a user experience and interface that is persona based so that collaboration is frictionless.
A couple of years ago we launched Helix Control-M, which is our SaaS offer for orchestration. We are seeing very strong demand from customers globally to consume orchestration as a SaaS model and we have plans for investing heavily in this consumption model for our customers.
- Basil Faruqui is the director of solutions marketing at BMC Software, which helps customers run and reinvent their businesses with open, scalable, and modular solutions to complex IT problems.
Want to learn more about cybersecurity and the cloud from industry leaders? Check out Cyber Security & Cloud Expo taking place in Amsterdam, California, and London. Explore other upcoming enterprise technology events and webinars powered by TechForge here.
