3PL Leader Achieves 100% API test automation coverage using JMeter

INDUSTRY

Logistics and Supply Chain

REVENUE

$1.2 Billion

This leading global third-party logistics provider offers a centralized transportation marketplace with a comprehensive multi-modal solutions portfolio — including TL, LTL, intermodal, and other capacity solutions to its customers. To meet their customer needs and growth goals in a rapidly changing market, the client deployed a wide range of applications/interfaces including mobile, web, windows, and automation backed by 100s of microservices.

Identifying the Challenge

To stay ahead of the competition and meet customer’s demand for new features/integrations, the client adopted a successful Agile / DevOps process for faster software development and release cycles. With the increased speed of development, and a heavy dependency on UI & manual testing, QA became a bottleneck inhibiting faster release cycles.

JMETER_IMAGE

Challenges faced:

40+ weekly deployments for various services & applications

Keeping up with their fast-paced development cycles 

Manual testing was slow and cumbersome

Changes in UI break the test cases and require constant updates

Overall delivery schedule & quality adversely affected as number of deployments rise

Insufficient time for testing applications/services

Prone to errors and bugs

Pushing new updates and features into production becoming a challenge

By identifying the challenges of manual testing and their heavy dependency UI testing, it was clear they needed to revamp their testing process and methodology. The next step was to implement new process changes and identify a tool that could alleviate some of their challenges and support their testing efforts as they continued to grow. The tool had to be easy to use while reducing their dependency on manual & UI automation with a higher emphasis on API automation.

Introducing: The Shift Left & Continuous Testing

As part of the agile development process, the client adopted short 2-week sprint cycles, putting Software development & QA teams under constant pressure to move faster. These fast release cycles resulted in a shorter QA window and testing while improving quality on each successive release.

The ‘shift left’ testing approach was introduced to accelerate testing cycles by injecting testing into each sprint to reduce test failures in later stages of development.

JMETER_IMAGE

Continuous Testing

The client’s software architecture changed dramatically over a period from Mono-micro, which facilitated faster releases. With the adoption of agile & DevOps, the client can move away from time-bounded releases to on-demand releases as the software is built & tested. Continuous Testing involves applying the methods and concepts of agile development to the testing and QA process, resulting in a more efficient testing process. Continuous testing allows “testing early, testing often” as we develop and acting as a “quality gate” as part of the build pipeline.

These new process changes provide-

Instant access to feedback by business stakeholders for more informed decision making

Ability to catch and fix bugs early in the sprint cycle

Elimination of defects at the source when they are easy to find, quick to fix, and cost less

Enforcement of “Quality gates” when “Continuous Testing” is part of the CI/CD process

Increased coverage which reduces risk and defect leakage

Increased efficiency and reduced technical debt

Need for API Testing

As the client shift to continuous testing, older approaches entirely focused on UI & manual testing become less effective. The traditional testing process followed by client, which relies heavily on UI & manual testing, needs to evolve to meet the needs of continuous testing and provide instant feedback to stakeholders. Continuous delivery can only be accomplished when the products go through continuous testing which requires automation. To achieve continuous testing, we need to shift our focus from the top of the test pyramid (UI automation) to the middle of the pyramid (API Automation).

API Testing was the secret to ensure the quality, performance, security, and reliability are achieved, resulting in faster release cycles. A successful continuous testing process requires the right tools and processes to be adopted across the organization.

Implementing API testing provides the following benefits- 

Improved stability and reliability

Continuous testing as part of CI/CD process

Greater test coverage

Extensibility for performance tests

Reduction in the overall time required for testing

Improved accuracy

Increased test coverage

Support for both manual and automated testing

It was evident that the antiquated testing processes and tools need a revamp to support the frequent addition of new features and releases. Without the proper tools in place, the process will fail. It was time to find a suitable API test automation tool that could accelerate the DevOps process and improve testing quality.

API Automation with JMeter

After an extensive evaluation of various commercial & Opensource tools to automate services, JMeter was chosen as the go-to tool for its simplicity and extensibility.

The following features made JMeter the best option for their needs-

Open-source tool

Highly extendable with pluggable Samplers & reusable fragments

Ability to load and performance test many different applications/server/protocols

Full-featured Test IDE that allows fast Test Plan recording (from Browsers or native applications), building and debugging

Extensive reporting with visual charts

Full multi-threading framework with concurrent execution

CLI Support can be integrated with DevOps pipelines for continuous testing

Easy to deploy and use without prior programming experience

Full multi-threading framework allows concurrent execution

Seamless integration with third-party open-source libraries such as Maven, Gradle, and Jenkins

JMeter tests can be executed behind a firewall or on the cloud using BlazeMeter for load testing

Ability to automate test executions and run the same test on-demand

Ability to support version control, incremental testing, and environment specific configurations

Capacity to combine different configuration fragments into one test configuration to comprehensively test functionality

Ability to create non-functional tests (performance, load, security, penetration, etc.)

The Robust Architecture of JMeter

JMETER_ROBUST-ARCHITECTURE

Technical Benefits: Accelerated DevOps cycle and more

By adopting API automation and “Shift-Left” & “Continuous Testing” methodologies, the client achieved faster release cycles with enhanced features with less dependency on manual testing.

100% automation coverage achieved around end-to-end regression workflows

Integrated APIs to YAML pipeline for continuous testing

Automated tests executed 24x7 as part of CI/CD pipelines

Shift-Left approach resulted in a 28% reduction in Software PRBs (Problem Records)

The ratio of problems to deployments decreased 63%

29% decrease in production bugs

The high/critical problem ratio dropped from 18% to 7%

Automated API testing resulted in a TTD reduction of 86% during the development/QA phase

Business Impact: Value addition in DevOps

API automation reduced the load on the software quality assurance teams so they could focus on value-added tasks. The product owners and software engineers experienced multiple technical and business benefits.

Accelerated product releases with increased deployment frequency, and quicker deliverables resulting in Faster time to market

Reduced recovery time with less impact or disruption on users

Low change/fail ratio

Software testing and business risks were resolved together

Better collaboration between the business, developers, and IT team

Improved security, reliability, and performance

Business stakeholders have the data to make more informed decisions

Defects are eliminated at the source when they are easy to find, quick to fix, and least costly to fix

Technologies that saved the day

apache-jmeter-tech-expertise

Read on

Video

Shift Left Testing Approach for 3PL Client​ | Emtec Digital

ROC LOGISTICS

Case Study

Leading Logistics and Supply Chain Provider effectively scales RPA initiatives utilizing Emtec Digital’s Robotics Operations Center

Top tech trends in Energy industry Blog_Thumbnail Image

Blog

Shifting Gears in the Logistics and Supply Chain Industry Post COVID-19

THUMBNAIL-SUPPLY-CHAIN-VISIBILITY-SOLUTIONS

Case Study

Custom Data Visualization Enables Supply Chain Leader With Intelligent Business Decision-Making

Sign up for insights