Multinational Energy Corporation Leverages DataOps to Enhance Organizational Data Insights

INDUSTRY

Oil & Gas

REVENUE

$20+ Billion

EMPLOYEES

5,000-10,000

The client is a multinational energy supplier to some of the world’s largest airlines, shipping companies, and power suppliers. With a widespread global network, their overseas operations span 100+ storage terminals, 3,000+ retail sites, and a presence at over 80 airports. With a mission to energize communities around the world, they are involved in the following services- 

Providing a range of infrastructure and energy solutions to retail and commercial customers

Developing sustainable and renewable energy solutions

To accomplish this mission, the client needed to make smarter use of their enterprise data to power business growth. As they were already utilizing Azure Data Factory, Emtec Digital’s DevOps experts helped build a robust solution that utilized their existing investments. 

Challenges: Lack of Data Insights, Time-consuming Feature Development, and Release Process

The client’s technology ecosystem consisted of humungous volumes of organizational data which was spread across the globe. Deriving meaningful insights from this data and leveraging them effectively for decision-making was a time-consuming process. There was an urgent need to consolidate data pertaining to people and processes across various locations. Their existing software suite was unable to meet their growth needs and needed a makeover. Challenges included: 

Lack of DevOps

The client faced challenges with creating multiple environments quickly and this hampered the pace of product releases. For ambitious organizations, products that evolve rapidly, tend to have a larger reach, and serve markets and customers better. The lack of DevOps impacted speed, competitive advantage, and the ability to innovate. The absence of a controlled environment on Production and missing feature development hampered business growth.

Inability to Develop and Deliver Software Efficiently and Accurately

The process to build software environments, fix bugs, make code changes and promote configuration from Development to Production environments took place manually. With such an effort-intensive and time-consuming process, the client’s internal teams required an enhanced process to create various software environments.

Issues Related to Code Promotion

The client had to ensure quick fixes in bugs and workflows with multiple teams working on several issues at the same time. There was a risk of uncooked code being promoted to the production environment.

Lack of Quality Checks

Azure Notebook had no check-in policy and no trace of code being overwritten or updated by users.

Solution: DataOps helped create more than 20 pipelines, analyze 3 TB of data across 10 enterprise apps

The solution for all these challenges was evident - a modernized and automated system for environment provisioning and release. In addition to the client’s existing Azure infrastructure, our experts suggested simplifying the use of Azure Data Factory in conjunction with the implementation of automated CI/CD pipelines in DevOps, supported by Emtec Digital’s own accelerators.

1. ADF Pipeline Automation and DevOps Governance 

Organized ADF nearly 20 pipelines for Non-Production and Production environments

Collaboration using feature branching in ADF

Integrated ADF with Azure Git and Azure Pipelines services

DevOps governance using Approval Flow on code promotion to Production

2. Monitoring and Reporting

Integrated ADF reports for pipeline progress and troubleshooting

Dashboards for easy access to ADF activities and runs

3. Automated Environment Provisioning using ARM Template

No manual intervention and potential for errors by resources

Time and effort savings

4. Data Bricks Notebook Promotion

Continuous tracking of every check-in and check-out process for better traceability

Deployment of Notebooks in higher environments - Dev →UAT →Prod using CI/CD pipelines in Azure DevOps

5. Integration of existing PowerBI reports to analyze data from almost 10 enterprise apps including Salesforce ERP, pipeline failures and monitor production ADF 

6. Azure dashboard to visualize nearly 3 TB of data analytics insights

7. Log Analytics to Log All Failures

Azure Data Analytics DevOps Architecture

Azure Data bricks code promotion and branching strategy

Azure databricks code promotion and branching strategy

thin

Technical Benefits

Automated the environment setup with the necessary resource provisioning (Infrastructure as Code) reduces environment creation time and increases accuracy

Feature branching enabled client team collaboration with selective feature promotion into production

Manual developer intervention removed to set up ADF pipelines in 6 higher environments

Enhanced traceability of changes in every aspect of Azure Data Bricks and Azure Data Factory

Seamless integration attained for development and production environments and streamlined process for data analytics

Only one/two clicks are now required for code promotion deployment into different environments with DevOps

Environment deployment speed cut down from one day to 30 minutes

Business Impact

Improved operational efficiency, accuracy, speed, and productivity with the implementation of DevOps

Rapid new feature development enables faster product releases and go-to-market initiatives

Minimal dependency on multiple teams with improved team coordination

One-click reports generated for global sites help with operational overview and time and effort savings

Enhanced business decision-making with speedy, timely, and accurate processing of organizational data

Technologies that saved the day

AZURE-DEVOPS
AZURE-DATABRICKS
AZURE DATA FACTORY
AZURE-LOG-ANALYTICS
Azure Blob Storage
POWER-BI

Want to accelerate ROI on your organizational data with DataOps?

Read on

THUMBNAIL - POWER BI ENERGY INDUSTRY CASE STUDY

Case Study

Actionable Intelligence | Power BI

Here’s how a global Energy Co. improved worker safety, productivity and terminal operations using…

THUMBNAIL-MICROSOFT POWER BI

Case Study

How to enhance Data Visualization and Analytics with Power BI

US Vet Care Chain Leverages MS Power BI for One Click Business Data Reporting Veterinary Industry…

Thumbnail-data-lake-solutions-case-study

Case Study

Enterprise Data Management | Data Lake Case Study

Learn how this multinational energy corporation streamlined the enterprise data aggregation proce…

Jmeter_Thumbnail image

Case Study

3PL Leader Achieves 100% API test automation coverage using JMeter

3pl leader automates API testing with JMeter and to improve test driven development by 86%.