A key part of a robust testing process is ensuring the quality of experience on the end-user device. But is it best to test with emulators and simulators, or with physical mobile devices?

A key part of a robust testing process is ensuring the quality of experience on the end-user device. Emulation and simulation provide flexibility in simulating many virtual devices in multiple scenarios, while device testing provides measurements of the actual experience on specific user handsets and devices from different manufacturers. So, is it best to test with emulators and simulators, or with physical mobile devices?

The distinction between Simulation and Emulation

There are pros and cons for both types of testing. First, it’s important to note the distinction between emulation and simulation. An emulator is used as a real node or Network Function (NF) in the network – in the case of Emblasoft, for example, it’s a virtual server that can be deployed alongside other NFs in a cloud environment. An emulator functions in the same manner as the real thing from a testing perspective. It copies the behaviour and performance of a network component by functionally ‘replacing’ it.

Conversely, a simulator – for example, on a network interface – mimics the behaviour of a network and its components in an abstract fashion to demonstrate specific behaviours and outcomes. Simulation generates the appearance of a behaviour or scenario that may not actually exist, while emulation recreates something that does.

In general, emulation is used for testing specific components of the mobile network for security, performance, or capacity, while simulation is applied to testing areas, interfaces, or protocols.

Testing with emulators and simulators is easy to provision, can scale massively, and offers automation. It is also more cost-effective. Simulation also provides active monitoring with virtual UE agents for customers who want to run higher frequencies and volumes of traffic, or to model behaviour for different user scenarios. They provide full control over multiple parameters – such as selecting MME and P-CSCF (to choose which IMS to test), as well as device settings and application selection. 

Injecting simulated traffic into the network enables the verification of different combinations of network entities and IMS networks, providing full control over all parameters. They offer improved control over the core network and enable proactive verification of the network. However, they cannot model the experience of using a physical mobile device and are limited in their ability to mimic the diverse range of devices with different operating systems, browsers, and hardware configurations.

Pros and cons of using physical mobile devices for testing

Conversely, using a physical mobile device for testing allows the precise end-user experience to be modelled, providing clear benefits from testing with physical devices. However, mobile devices are expensive, which makes testing at scale costly and potentially out of reach for smaller companies. Multiple devices are needed to test the broad range of different models and brands available, so costs can easily rise.

Mobile phones also need to be managed and updated, and as mentioned it is a challenge to scale and test en masse. Furthermore, tests are limited to a low level of call volumes (for example, every 5 mins, etc), making modelling traffic volumes problematic. If you only use mobile phones for testing, you are missing out on: 

• Frequency
• Volume
• Control

In summary, emulators and simulators are easy to provision, scale massively, offer automation, and provide full control over test parameters. On the other hand, mobile devices mean that you can drill down into the actual (not emulated) experience of an end user. It means that a combination of both is the ideal scenario for a comprehensive testing program, allowing comparison of the results from each type of test. In an ideal world, both types of tests should be used.

Emblasoft Emulators and Evolver Simulator

To close the gap, it’s important to use simulators and emulators as part of your service and network test strategy. Emblasoft offers both. Our emulators include functions such as:

• PCRF/PCF Emulator
• HSS (Home Subscriber Service) Emulator
• NRF Emulator

Our PCRF / PCF Emulator is designed for performance and stability testing based on the Sd Diameter Interface. The HSS Emulator is designed for testing performance and stability using a subset of the Sh, Zh, Zx and SWx Diameter Interfaces.

Our NRF Emulator allows MNOs and vendors to emulate and test NF registration and to create a test environment with servers that support performance and functional testing. It can test registration and functionality for different characteristics and QoS demands. This can be done at scale and with multiple NFs, so that a diverse, multi-vendor environment can be validated and tested. Other options are available.

Emblasoft Evolver, meanwhile, enables Simulation. For example, Evolver can terminate at the N4 interface when it’s being used to test the UPF. This means that, from the UPF perspective, the SMF (as well as the functions behind the SMF) are being simulated, not emulated. As Evolver supports multiple interfaces, it means that specified tests and testing performance with different traffic load mixes can be simulated in conjunction with emulation testing.

A mix of Emulation and Simulation alongside mobile device testing is the ideal scenario

Similarly, Evolver’s Active Monitoring solution is based on a complete, distributed architecture, extending from monitoring agents (UEs) to the execution platform and control centre. Configurable software-based agents generate user traffic and capture metrics regarding the QoS delivered, for multiple applications and services, across all technology domains.

Emblasoft’s comprehensive range of emulation and simulation testing solutions, which can be provided as an end-to-end solution or as standalone services, ensures that vendors and MNOs can rigorously test their network environment, from the 5G core to NFs and everything in between.

In conclusion, it’s essential to use emulation and simulation in conjunction with physical mobile devices to ensure you get the full picture. That way you can test, measure, and monitor QoE throughout the entire service chain. Contact us today to find out how we can help you to solve your testing challenges.