Complete set of equipment for drive tests, per each team:
Laptop with NEMO Outdoor, including POLQA options for Voice
testing
9 up-to-date handsets and SIM cards for voice, SMS and data testing
GPS receiver
A technical board, to place testing devices, with USB cables and
fixed HUB, and a charging system to plug on the vehicle for drive tests
Additional equipment if needed for specific measurements (walktests,
scanner, signal attenuators, …)
Complete drive-test solution, for each team:
A vehicle (4*4 SUV type), customized with TRA’s logo , with a
rooftop box to contain the handsets in outdoor situation
A confirmed field engineer for drive test.
DRIVE TEST
QoS weekly drive test will be performed through drive test over all areas in the Kingdom of Bahrain, including its main roads and motorways;
Test methodology follows ITU ref P.800 (Mean Opinion Score test for voice specification)
The technical solution, including measurements tools and software, is fully automated with the POLQA algorithm, which is currently the reference for the evaluation of Voice Quality, using NEMO Outdoor.
A measurement is a call attempt followed by a 2-min conversation.
Calls are 100% Mobile to Mobile (MTM) on their own network, sender and receiver being co-localized on the same location.
If a Call is set-up on first attempt and held for 2 minutes without drop, its quality is evaluated according to a POLQA score threshold, from perfect too bad.
Evaluation of quality through POLQA is based on the definition of MOS (Mean Opinion Score) in UIT-T P.800 recommendation, which gives the following 5 levels scale for score:
|
Perceived quality |
Listening effort |
MOS score |
|
Excellent |
no effort |
5 |
|
Good |
No specific effort |
4 |
|
Fair |
Moderate effort |
3 |
|
Poor |
Huge effort |
2 |
|
Bad |
Incomprehensible despite all possible efforts |
1 |
The Current 4 level scale used for manual quality tests (m-MOS) is:
|
Note |
Description |
|
Perfect |
Tester doesn’t notice any defect |
|
Fair |
One defect occurs while the conversation goes on uninterrupted |
|
Poor |
The natural flow of the conversation is altered. Tester has to repeat himself |
|
Bad |
The defect is so strong that conversation cannot be proceeding. |
A test consists of attempting to send an SMS message, measuring delay for reception and verifying its integrity, on each of the networks tested.
SMS are measured the same Nemo session and the same devices as Voice tests, 100% on their own network. Cross network SMS can also be tested if required by TRA
Following Data services are measured automatically with Nemo Outdoor:
· HTTP Data Transfer (upload and download)
· Web browsing
· Video browsing
· Over The Top and Social Media Apps consumer Experience
On each network: downloading and uploading a file during 10 seconds
Recording the total size of file that could be transferred to compute throughput
Size of file will to be determined, at least 1GB
This allows the same methodology regardless of the network * TO = Time Out
Test servers with sufficient bandwidth (100Mb/s)
On each network, a measurement consists of downloading one of the 10 most visited public homepages and one page from each Operator, taking note of completion time, errors on the page if any, with a 30 seconds timeout.
The selection of URL is the same as 2019 QoS audit:
|
http://www.amazon.com |
|
http://www.apple.com |
|
http://www.expatriates.com |
|
http://www.facebook.com |
|
http://www.google.com |
|
http://www.instagram.com |
|
http://www.microsoft.com |
|
http://www.youtube.com |
Figure 3 : List of webpages tested.
Streaming Measurements are carried out by assessing the quality of popular YouTube videos with smartphones in order to represent closely as possible the customer experience.
2 different quality level are tested: 720p and 1080p.
Evaluation of the quality of service on social networks is measured automatically with NEMO Outdoor.
· Facebook : feed post
· WhatsApp : outgoing call and messaging
· Instagram : Get Self Feed
Objectives:
· Measure the outdoor coverage of the 3 Mobile Operators; Batelco, Viva and Zain, via an accessibility test, for the following technologies : 3G, 4G and 5G;
· Establish for each operator a direct correlation between the number of households covered and the percentage of the population, resulting directly from such coverage
· Validate the coverage maps of each Mobile Operator against the outdoor coverage observed during the measurements
Measurements are performed using Nemo outdoor with standard handsets.
The following protocols describe all settings used by Directique for the coverage audits in Bahrain for TRA (since 2014).
Accessibility Data tests are performed with:
· 1 set in 2G/3G/4G auto connect, to represent LTE’s users
· 1 set in 2G/3G auto connect, to represent non LTE’s users
· 1 set of 5G devices, to represent 5G users
The tests consist in downloading a file of the same size every 30 seconds all along the coverage drive:
- For 3G/4G : Download of a small file - 512 bytes (.bin),
- For 5G : Download of a big file – 1GB – during 2 seconds, in order to give time to the handset to engage the transfer in 5G technology.
|
KPIs |
Definition |
|
SHC (Set-up and held for 2 min calls) |
% of calls set-up and held for 2 minutes. Call set-up on first attempt and held for 2 minutes without drop. Rate is based on the total sample |
|
GQR (Good quality rate) |
% of calls set-up held for 2 minutes and without a POLQA sample below the threshold. Calls excluded = failed on first attempt, dropped before 2 minutes, or with a POLQA sample below the threshold Rate is based on the total sample |
|
KPIs |
Definition |
|
RS10 (% of SMS received within 10 sec.) |
SMS not refused when sent out and received within 10 seconds without being altered |
|
KPIs |
Definition |
|
WEB10 : % of successful data transfers within 10 seconds |
Successful page loading within 10s. Rate based on total sample |
|
WEB5 : % of successful data transfers within 5 seconds |
Successful page loading within 5s. Rate based on total sample |
|
DL/UL : % of transfers with a throughput > 2Mbps |
Successful data transfer with a speed > 2Mbps that allow the most common experiences |
|
DL/UL : Average Throughput |
Average throughput once connected, applied only to successful data transfers |
|
DL/UL : Best Throughput |
Best throughput recorded for a data transfer measurement |
|
Streaming - LHV : % of videos set-up and held for 2 min |
Video launched on first attempt, and held for 2 min without drop |
|
KPIs |
Definition |
|
Coverage - % of successful access attempts |
Rate of successful DATA packets attempts Indicator is based on the total number of attempts |
Directique is a specialized consulting firm entirely dedicated to mobile telecom test and measurements since 1993. With a great range of clients worldwide, including:
Directique has successfully carried out all TRA’s quality of service and coverage audit since 2009.
As early as 1996 on behalf of the French regulator, Directique designed test methods to capture customer perspectives on mobile services, and the company was since regularly selected to carry out the Authority’s annual benchmarks (and every year from 2003 to 2018).
Directique is the main provider to operators, with a share of 50% of its home market.
Directique is known as a reference in the design and validation of measurement protocols for coverage auditing, and is surveying its home market coverage every year since 2003.