Smart Plug Scoring

To provide a data-driven, objective basis for the comparison of smart plug performance in our reviews, the following set of measurable, repeatable testing procedures have been designed. These tests are based on the key factors that are important for a smart plug to fulfill the roles that owners would expect of them in areas such as responsiveness, energy monitoring, and reliability.

The measurements are based on real world usage scenarios, tested while installed in a real home setting. All of our usual editorial review guidelines still apply.

The scores derived from these tests are presented in product review, comparison, and buyers guide articles with color coded indicators as follows:

Smart Plug Performance Categories

Smart Plugs are evaluated across 5 key performance categories. Each of these has specifically defined scoring criteria detailed in the relevant section below. The overarching categories are:

1. Setup Experience

2. Responsiveness

3. Energy Monitoring

4. Reliability

5. Privacy and Security

6. Internet Usage

7. App Experience

Setup Experience

The first hurdle for most smart devices is getting them configured on the home network. This category rates the setup implementation of the smart plug based on a points system as follows:

Setup process

• Does not require an app account for setup - 1 point

• Adding new device is intuitive and easily initiated - 1 point

• Setup process steps are clear and assume nothing - 3 points

• Setup steps include diagrams or animations for clarity - 2 point

Network discovery (choose one)

• Manual network configuration - 0 points

• Manually joined ad-hoc access point for initial connection - 1 point

• Auto joined ad-hoc access point for initial connection - 2 points

• Out-of-band auto-discovery (Thread, Bluetooth, ZigBee etc) - 4 points

Setup Success (choose one)

• Setup successful on first attempt - 4 points

• Setup required 1 retry - 2 points

• Setup requires 2 retries - 1 point

• Setup requires more than 2 resets/retries - 0 points

Scoring is the total points divided by the total possible of 15 expressed out of 10.

Responsiveness

This test measures how quickly the smart plug can respond to on/off requests. Timings will be measured for native app control, third party app control (eg. Alexa), voice command, and automation triggers. A maximum time of 10 seconds is allowed for any score, which accounts for longer run times of voice and automation triggers and provides favorable weighting for faster response times overall.

To account for variability in network (and internet where applicable) performance 5 measurements will be taken on different days for each of the 4 types.

Scoring is the average measured response time as a percentage of the maximum acceptable time of 10 seconds, allowing for measuring inaccuracies of +/- 0.5 seconds.

Energy Monitoring

To determine the quality and usability of the provided energy monitoring function, points will be awarded across a number of categories. Scoring is the total points converted to a percentage of the maximum of 16, expressed out of 10.

Data retention (cumulative)

• 7 days = 1 point

• 30 days = 1 points

• 180+ days = 2 points

• Data can be accessed by third party apps - 1 point

• Archive/export functionality - 1 point

• Data Reset Option - 1 point

Usability (cumulative)

• Hourly data inspection - 1 point

• Date search - 1 point

• Spot data check - 1 point

• Swipe to scroll - 1 point

Cost Calculation (cumulative)

• Configurable unit cost for calculation - 1 point

• Current consumption - 1 point

• Total consumption - 1 point

• Cost/consumption projection - 1 point

• Multiple energy costs for time of use - 1 point

Reliability

This test attempts to quantify how often the device is unavailable, drops off the network, or becomes unresponsive. After the initial responsiveness test cycle of 5 days, the smart plug will be used on an infrequent basis for 30 days to watch for any availability drop. Dropouts will be recorded.

Each incident will be recorded as follows:

• No response, but retry was successful = 0.5

• Reboot required = 1

• Full reset required = 5

Scoring is the total dropout count drop subtracted from 30, expressed out of 10.

Internet Usage

This gauges the ‘chatiness’ of the device, that is how much data it sends over the internet. While this data is typically encrypted and cannot be assessed, and excessive amount of data communication is problematic as a smart device should not need to send much at all. Measurements will be recorded weekly for 4 weeks for both upload and download traffic.

Scoring will be the amount of internet traffic recorded over a 4 week test period subtracted from 100MB, expressed as a score out of 10.

Privacy and Security

These factors represent the essential components of a viable security model for a smart app, with higher points awarded for design that promotes security in an accessible way for typical users. Points are awarded for each key factor, converted to a percentage of the maximum possible score of 12, expressed out of 10.

Authentication (choose one)

• 4 points - Device-based auth token with long life.

• 3 point - Periodic Biometric auth on open.

• 2 points - Periodic manual password re-entry on open.

• 1 point - Password re-entry on every open.

• 0 points - No authentication.

Access Management (cumulative)

• 1 point - Device sharing between accounts via invite (avoids shared password).

• 1 point- Authenticated device/user list.

• 1 point - Ability to revoke access to device/user.

• 1 point - Two-factor authentication is available.

• 1 point - Two-factor authentication is mandatory.

Firmware Updates (choose one)

• 1 point - firmware updates are clearly notified in app.

• 2 point - firmware updates are advertised by push notification.

• 3 points - firmware updates are automatically installed without intervention.

App Experience

App usability is typically a more subjective area, as such the following factors have been identified as aspects that objectively promote a usable and functional user experience. Other design choices will appeal to different users, so cannot be scored, but will be noted in the review content. Points are awarded for each key factor, converted to a percentage of the maximum possible score of 9, expressed out of 10.

Feature accessibility

• 4 points - Device features are logically organized and easy to find.

• 1 point - Non-standard/new features have explanatory text.

• 1 point - Non-standard/new features have additional explanatory diagrams/animations.

• 1 point - New features are announced clearly in the interface.

• 1 point - Support documentation is contextually linked where significant explanation may be required.

• 1 point - Support documentation is easily located.

Quality Issues

• Lose 1 point - minor bugs observed that cause cosmetic or slight nuisance that doesn’t impact usage.

• Lose 2 points - significant bugs observed that obstruct usage and cause frustration.

• Lose 4 points - major bugs observed that cannot be worked around.