Aricent recently concluded a series of webinars on “Cloud-based Telecom Testing: Why and How”. The webinars focused on the advantages and strategies for cloud-based testing and highlighted our powerful Software Tool for Automated Testing Environment (STATE) which has been ported to the cloud to enable LTE testing. Even though the Q&A sessions were quite lengthy, we had more questions left to be answered. Here are the answers to all the questions that we couldn't get to during the webinars.
Also, the webinar recording is available for the next 18 months and can be viewed online.
I’m an OEM vendor and have the entire infrastructure already maintained. How can cloud based-testing help me?
In case, the lab environment is already created with the entire infrastructure, then you have to carefully evaluate whether to move to the cloud. There are certain aspects which you can consider during evaluation like OpEx, technical staff for support, anytime anywhere accessibility, scope of enhancing the hardware/software, and scalability/effort consumption. Apart from the expenditure, you need to analyze the benefits that cloud based testing offers over the traditional lab test setup.
I’m looking for testing the SUT using Test Setup-I that was discussed. What details should be shared by me to enable testing?
If you want to test the SUT through the cloud-based testing tool, then the following details need to be shared in order to enable connectivity:
- SUT IP and it’s communication port
- SUT’s call related basic configuration (such as PLMN ID, IMSI etc.)
- The desired mode for operating on the cloud (http, https, ssh, rdp, vlc)
Apart from providing these details, one should take care of the following at the lab premises:
- Move the SUT towards DMZ for security
- Open the firewall ports to enable connectivity
How secure is the cloud-based testing environment? What are the security measures one should take before adopting a cloud-based test environment?
This question can be answered in two ways with respect to the test setups:
Test Setup-I is very well secured because your SUT or software enablers is inside your network that’s protected under the DMZ and you are just accessing the test framework available on the cloud for test execution.
For choosing Test Setup-II one must ensure the following before migrating the SUTs to the cloud:
- Ensure the visibility of your SUT in the multi tenant environment
- Analyze security characteristics of the cloud service provider.
- Ability to control the accessibility towards your servers based on the profiles
- Whether encrypting your data is mandatory
Also, when it comes to security, different companies have different security mechanisms to protect their software enablers and organizations considering cloud-based services must understand the associated risks and ensure appropriate visibility. However, the cloud computing environment is secure and there are different types of security measures taken at different levels, like infrastructure, platform, and application levels. Also, there are organizations which provide software’s that protect your software enablers on the cloud.
Does your STATE test framework support the load/performance testing?
The STATE tool supports different typing of testing phases like feature, functional, regression, system testing, and system integration testing etc. Although STATE is not currently ready for load/performance testing, it can be used when the KPIs are not in the higher range. Also, STATE currently does not support multi-threading.
Has your test framework been validated by any customers?
We have LTE frameworks for eNB, SGW, PGW, MME, and PCRF. Our test framework has been validated by our internal customers on the LTE domain. Also, the STATE tool and the frameworks have been used extensively by the external customers for SS7, MAP, and SIP conformance testing.
What is the difference between the cloud and virtualization?
Virtualization is a technique that allows enterprises to create several virtual servers on a single physical server and these servers are located inside the enterprises lab. The created virtual servers can run on different operating systems simultaneously. In virtualizations, the infrastructure/platform/software applications are managed by the enterprises themselves.
The cloud computing environment is non-local. The infrastructure/platform/applications are provided by the vendor and these are accessed through the internet. Cloud computing takes advantage of virtualization technologies.
How long does it normally take to create a test environment on the cloud?
The creation of a test environment on the cloud is highly dependent on the size of the test environment you are planning to create. If you are considering adopting the Test Setup-I, then the following activities need to be done:
- Moving the SUT to DMZ environment
- Co-ordinate with IT to enable connectivity with the cloud
- Configuring the cloud test tool according to your SUT configurations
Normally, it takes a few days to setup the test environment with a single SUT in the lab environment.
For Test Setup-II, the SUT is available on the cloud along with the test framework. One should consider the following after finalizing the strategic cloud partner:
1. Installation of your SUT on the cloud:
- Creation of the servers
- Installation of the OS and application software
- Configuration of the SUT
2. Configuring the cloud test tool according to your SUT configurations
3. Co-ordinate with IT to enable connectivity with the cloud
In this case, it normally takes a few weeks to setup the test environment on the cloud.
What different phase of testing or telecom domain testing can one do with STATE?
Using STATE one can achieve unit, functional, feature, regression, system, and system integration testing. Currently, STATE does not support load/performance testing, but can be used when the KPIs are not in the higher range.
Won't delays on the internet impact the testing? Eg. Timers on the SUT could expire due to these network delays in delivering messages from the test tool and vice versa.
In order to perform effective cloud-based testing, we suggest having higher internet bandwidth (dedicated 2 Mbps or higher). We have tested an SGW at our premises with the test tool at the cloud and communication is based on UDP/IP and TCP/IP. During the course of testing, we have not faced any issues related to the expiry of timers due to the delays caused by the internet.
What is the capacity of simultaneous sessions, for example, can STATE support 2,000 simultaneous sessions coming from the cloud to the SUT?
STATE is mainly intended to support functional testing. It can send calls sequentially. Simultaneous call sessions are not supported by the tool.
How do you detect when to give inputs for the scenarios?
In cloud-based testing everything is in a software-based environment. In this environment you remotely connect to the lab network through your desktop. For example, you are testing a SIP based User Equipment at the lab premises through the cloud-based testing tool. In this case, for incoming call testing, the cloud tool simulates the peer entity and sends an INVITE message to the IP phone. By receiving the INVITE message, the phone will get alerted. In the case of outbound call testing, the IP phone at your premises initiates the call towards the peer entity (ex.IP phone or proxy server) simulated on the cloud. In this case the script should have been started in the test tool prior to initializing the call from the IP Phone. Depending on the scenario, the user or the test tool will alert the user to provide inputs for the scenario.
POC/points to consider/test confidence etc. are very basic things of discussion. Will you list out more usability scenarios?
The usability scenarios may vary from requirement to requirement. For example, in our case, we had to test the SGW located at the lab with the test tool available on the cloud. The SGW supports GTPc-v2, GTPc-v1, Diameter, and GTP-U protocols over UDP/IP or TCP/IP. According to the specifications there are standards ports through which these entities communicate with each other.
- For providing more security to the SGW we moved it to DMZ.
- Unblocked ports at our firewall (for administrative and call establishment purpose)
- Ports blocked by the cloud vendor were opened on request.
- To gain confidence in cloud-based testing we ran all the top priority test cases.
- Validated our security policies by executing some intrusive tests.
If you still need additional clarifications, please feel free to contact us.
Will using VMware for SUT's work in the cloud?
We think, by saying VMware, you mean the SUTs are created using the VCentre/VSphere application. In this context, yes, the SUTs available in the virtual machines can be exposed towards the tool available in the cloud and testing can be done. However, prior to exposing the SUTs, one should move these SUTs to the DMZ for providing extra security.
How do you deal with testing and test automation when physical telecom devices such as routers and switches are required?
You should use Test Setup-I, in which only the test tool is ported on the cloud and the remaining physical devices such as routers and switches are available at the lab premises. In this setup, the user accesses the test tool that is connected to the physical devices in the lab over internet, so one can perform testing. This type of testing model may be supported in Test Setup-II, if the cloud service provider can facilitates it.
Is accessing the cloud server through RDP/HTTP secure?
The server supports different modes of communication like RDP, HTTP, SSH, VLC, and HTTPS. The user can choose the communication mode as per need. Predominantly, users are provided with a User ID and secure password through which they can login to the https/http address and access the tool and test framework on the cloud for test execution. RDP may be used for administrative purpose and it is not necessarily shared with the end user. Most recently Microsoft announced vulnerability in RDP and provided the patch for its Windows operating system. This is one of the areas to be improved in the cloud space.
Can your STATE tool be leveraged by enterprises that have an interest in testing IPv6 on embedded systems over cellular technology?
Yes, STATE supports IPv6 and can run on IPv6 machines. Also, it can establish the IP connection with the SUT. However, for end-to-end IPv6 testing, we believe that the whole test environment (SUT, test tool, intermediate networking devices such as routers and switches etc.) should support IPv6. As such, STATE does not limit testing IPv6 on embedded systems over cellular technology. In case you need more information on this, you can get back to us with the details related to your SUT and what type of IPv6 testing you wish to perform.
How do you deal with parallel releases being done at the same time in terms of pulling down the environment and setting it up?
One can leverage the cloud’s flexibility to add, delete, and modify the test environment with minimal effort Also, one can subscribe or unsubscribe based on the need. However, when parallel releases are being executed, we understand that the test beds are constantly utilized, and in this case we do not see any requirement for pulling down the test environment and setting it up again.
How will you do performance testing for cases where performance matters, for example SIP stacks etc.?
One can opt for Test Setup-II in which both SUT and test tools are ported on the cloud for executing performance related scenarios. Since, both SUT and tool will be connected on the same VLAN the signaling and data pipe can be created inside the cloud environment itself. The testing is not limited to SIP stacks alone and any SUT can be validated for performance, if both the SUT and the load/performance test tool are available on the cloud environment. If Test Setup-I is to be used for performance testing, then significant internet bandwidth is required between the cloud-based tool and the SUT at the lab.
What kind of BSS are you able to test? What about BSCS?
The STATE tool is a generic tool that can enable different types of testing and technologies. So far we have not utilized this tool in our BSS testing engagements. However, we have simulated application layer protocols like MAP (Mobile Application Part) and other SS7 protocols using STATE and tested a real HLR.
What are some of the cons of cloud-based telecom testing?
Following are some of cons of cloud-based telecom testing, if proper planning is not done:
- The reliability of the vendor who provides the service
- A dedicated higher bandwidth internet to avoid delays in message exchange
- Extra security measures to be taken to protect the software enablers on the cloud
What is the difference between Platform and Infrastructure as a service?
Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) are two different service models offered by cloud service vendors. In the IaaS, the service provider offers just the hardware over the cloud, on which one can install other necessary software to host or run applications. The user has full control over the infrastructure such as servers, operating systems, and security related features. In the PaaS service model, the service provider offers its platform on which one can create applications to run specifically on that platform. The user has no control over the hardware, operating system, and security related features.
How would the Hybrid deployment mode be setup?
Hybrid cloud is a combination of Private and Public cloud. The private cloud holders can engage with a cloud service provider, who offers and supports the hybrid cloud, to deploy the hybrid cloud environment.
It was said that Test Setup II will enable non-functional tests. Does Test Setup I prevent it?
Setup I doesn’t prevent non-functional testing. However, it is not an optimized and cost effective solution because:
- The number of hops for the data transacted between the cloud and SUT may not give the expected KPIs.
- It requires more dedicated internet bandwidth between the test tool on the cloud and the SUT at the lab premises. Also, more data sent and received towards the cloud increases subscription costs.
Hence, we suggest going for Setup II instead of Setup I for non-functional testing.
On what basis have you compared test-bed time as 2 days on cloud but 5 days in the traditional environment? Please provide supporting data with an example. How does setup time get reduced in cloud based testing from 5 days to 2 days. Are these costs daily or hourly rates?
In our case, we had to deploy a server in the lab environment with the necessary software applications running on it. The effort mentioned is based on our requirements for creating the test environment. This may vary for other requirements.
The traditional test environment setup time is calculated based on some of the following:
- Procurement of the servers, software
- Lab space identification, power requirement
- Installation of the hardware, software (operating system, other necessary software to run the test framework)
- Co-ordination with IT to get the IP addresses and enable connectivity
- Commissioning the test tool to start test execution
These points may vary as per the process followed in a company. Each of these activities mentioned above causes delay and thus it takes more time to setup the test environment. In the cloud environment, setting up an infrastructure and installation of the operating system, IP address assignment, and other necessary software installations can be completed in just a few clicks. Thus, it saves considerable effort in setting up the test environment.
Regarding costs, cloud service providers have many innovative costing models based on the subscription. One can choose the costing model as per need.
Is there any guide for STATE illustrating the features that it supports or can support?
Yes, we do have documents of the STATE user manual and a feature guide that illustrates the features and supporting capabilities. These documents are included in the STATE delivery package.
Cloud services are used over an unreliable Internet connection, hence testing may be frustrating if the connection keeps breaking during a lengthy test case. What is your suggestion on it?
Yes, we agree. The internet is the backbone for performing cloud-based testing. But we assume that all corporate networks will have a reliable dedicated internet connection. Thus the issue of unreliable internet connection is negligible.
Assume I have an "abc" cloud-based system with SQL. Tomorrow I would like to port it to another cloud provider. Do we have a seamless data migration approach for moving the applications?
As we said during the webinar, seamless migration between cloud vendors is at a nascent stage. Recently, few vendors have started offering the seamless migration of data across vendors. This needs to be further evaluated.
How can we be sure that the cloud-based testing results will represent real life scenarios, and not just a simulation?
Cloud-based testing differs from traditional testing only in the fact that in cloud-based testing the test equipment are managed remotely whereas the in traditional testing they are managed locally. Generally, testing is accomplished using either the test simulators or real nodes. The simulators provide the flexibility to create more realistic and inopportune scenarios to test the SUT in an effective manner. Cloud-based testing provides cost effective solutions by leveraging its capability to host the simulators or a real node on the cloud. Using cloud-based testing one can create real life scenarios either using simulators or real nodes.