Electronic Design
  • Resources
  • Directory
  • Webinars
  • CAD Models
  • Video
  • Blogs
  • More Publications
  • Advertise
    • Search
  • Top Stories
  • Tech Topics
  • Analog
  • Power
  • Embedded
  • Test
  • AI / ML
  • Automotive
  • Data Sheets
  • Topics
    - TechXchange Topics --- Markets --AutomotiveAutomation-- Technologies --AnalogPowerTest & MeasurementEmbedded
    Resources
    Electronic Design ResourcesTop Stories of the WeekNew ProductsKit Close-UpElectronic Design LibrarySearch Data SheetsCompany DirectoryBlogsContribute
    Members
    ContentBenefitsSubscribeDigital editions
    Advertise
    https://www.facebook.com/ElectronicDesign
    https://www.linkedin.com/groups/4210549/
    https://twitter.com/ElectronicDesgn
    https://www.youtube.com/channel/UCXKEiQ9dob20rIqTA7ONfJg
    Electronicdesign Com Content Content 73765 73765 Fig Sm
    1. Technologies
    2. Embedded

    Know Your Benchmarks Before You Make Comparisons

    April 17, 2012
    What do benchmarks and surveys have in common? They can be good, bad or ugly.
    William G. Wong

    A Polk survey has touted that “only” 35% of hybrid car owners would buy another. Unfortunately, the details of the survey are not available, but Polk is in the business of selling information so tidbits get tossed around.

    My wife and I own hybrids. My wife likes her Honda Civic hybrid, while I have a Toyota Prius (see the figure). She likes the smaller size, and I like the ability to run all-electric. It’s handy when I’m sitting in traffic or driving around the neighborhood. It’s one of the smoothest cars I have ever driven.

    Electronicdesign Com Content Content 73765 73765 Fig Sm
    The 2012 Toyota Prius looks slick but I’ll probably have my Winter Gray 2010 model for a few more years yet. At that point, another hybrid or electric will be the replacement.

    The problem with the survey results is that it’s difficult to determine what they mean because we don’t know what questions were asked and what alternatives were provided.

    Lies, Damn Lies and Statistics

    Cars are always a hot topic with designers, programmers, and engineers. So are other measurement issues like benchmarks, which are a lot like surveys. The idea is to present useful numbers that allow people to make comparisons that will affect future decisions.

    Many years ago when PC Magazine was still a print publication, I was the director of PC Labs, which was known for its benchmarks. This was decades ago when comparing processors, printers, and graphics cards was relatively easy. We were able to compare related processors, and processor benchmark results were often a function of clock rate.

    Benchmark results and comparisons are useful when it’s possible to extrapolate from these values. It helps to know what was tested so the extrapolation doesn’t go off to places it shouldn’t. This is the problem with statistics in general. One wrong assumption and all bets are off.

    Still, benchmarks and surveys can provide useful information. The more one knows about what is being measured and why, the better.

    One challenge at PC Labs was reducing the number of results while retaining useful information. Everyone wanted a single number as a result to make their job of evaluating the items being tested easier. But single numbers tend to make results misleading when a lot of variables come into play.

    Embedded Benchmarks

    Computer benchmarks have been around for ages, and they continue to be useful when they’re used properly. One source for the embedded market is EEMBC, originally known as the Embedded Microprocessor Benchmark Consortium.

    EEMBC benchmarks are highly regarded and numerous, allowing people to examine key aspects of a system design. One of the most commonly highlighted benchmarks, the CoreMark, actually generates a single number. It’s designed for single-core processors and written in C, enabling it to be relatively portable.

    CoreMark does a good job of generating comparable results when comparing two microcontrollers with the same architecture using the same compiler. Change compilers or architectures, and the comparisons become more tenuous. Of course, this doesn’t stop marketing departments from touting better CoreMark values.

    Unfortunately, even the simple CoreMark tests the architecture, compiler, and a particular instance of the chip. I would consider differences of an order of magnitude important, but smaller fractions could easily be due to other factors. Still, using benchmark and survey results with care and understanding can be very useful. Likewise, the plethora of benchmarks from EEMBC can be very useful.

    If you want more complex benchmarks than EEMBC’s, try Android AndEBench and the GrinderBench for Java, which are two examples of system-oriented tests. Then there are processor-oriented benchmarks like the FPBench for floating point and DENBench for digital entertainment products. There is even a power usage benchmark, EnergyBench. Still, EEMBC’s benchmarks provide consistency and lots of results.

    The challenge is deciding what numbers are going to be worthwhile for you in addition to knowing what will be relevant. Unless you’re generating your own benchmarks, you will likely need to compromise on both the tests and the results.

    My wife and I are empty nesters. We went through a Toyota and Honda van when the kids were at home, but we didn’t even consider buying another one when we were downsizing. An electric car may be in our future, but that will be a few years from now. We tend to drive our cars until they don’t run anymore, and the Civic and Prius have a long life ahead of them.

    Benchmarks and surveys aren’t always about the answers. Sometimes they’re about asking the right questions.

    Continue Reading

    First Takeaways from Our 2023 Career & Salary Survey

    Airborne Electromagnetic Survey Offers Birdseye View of Geographic Details

    Sponsored Recommendations

    Designing automotive-grade camera-based mirror systems

    Dec. 2, 2023

    Design security cameras and other low-power smart cameras with AI vision processors

    Dec. 2, 2023

    Automotive 1 TOPS vision SoC with RGB-IR ISP for 1-2 cameras, driver monitoring, dashcams

    Dec. 2, 2023

    AM62A starter kit for edge AI, vision, analytics and general purpose processors

    Dec. 2, 2023

    Comments

    To join the conversation, and become an exclusive member of Electronic Design, create an account today!

    I already have an account

    New

    Super-Slim Wireless, Self-Powered “Sticker” Measures Contact Force

    Commercial-Ready Route in Texas for Driverless Trucks Planned for Late 2024

    Running Background Checks on Today’s Celebrity Scientists

    Most Read

    Autonomous Robot Delivers Open Liquids Container Without Spilling a Drop

    Solve Augmented-Reality Display Challenges with Laser Beam Scanning

    The Fastest Way To Connect To The Cloud…Learn About IoTConnect


    Sponsored

    Tackling the EMI challenge

    TTI Hybrid and Electric Vehicles Line Card

    1 TOPS vision SoC with RGB-IR ISP for 1-2 cameras, low-power, video surveillance, retail automation

    Electronic Design
    https://www.facebook.com/ElectronicDesign
    https://www.linkedin.com/groups/4210549/
    https://twitter.com/ElectronicDesgn
    https://www.youtube.com/channel/UCXKEiQ9dob20rIqTA7ONfJg
    • About Us
    • Contact Us
    • Advertise
    • Do Not Sell or Share
    • Privacy & Cookie Policy
    • Terms of Service
    © 2023 Endeavor Business Media, LLC. All rights reserved.
    Endeavor Business Media Logo