AppleInsider got the motherlode. After several years of back and forth debates about its testing procedures, Consumer Reports magazine invited the online publication to tour their facilities in New York. On the surface, you’d think the editorial stuff would be putting on their best face to get favorable coverage.
And maybe they will. AppleInsider has only published the first part of the story, and there are apt to be far more revelations about CR’s test facilities and the potential shortcomings in the next part.
Now we all know about the concerns: CR finds problems, or potential problems, with Apple gear. Sometimes the story never changes, sometimes it does. But the entire test process may be a matter of concern.
Let’s take the recent review that pits Apple’s HomePod against a high-end Google Home Max, which sells for $400 and the Sonos One. In this comparison, “Overall the sound of the HomePod was a bit muddy compared with what the Sonos One and Google Home Max delivered.”
All right, CR is entitled to its preferences and its test procedures, but let's take a brief look at what AppleInsider reveals about them.
So we all know CR claims to have a test panel that listens to speakers set up in a special room that, from the front at least, comes across as a crowded audio dealer with loads of gear stacked up one against another. Is that the ideal setup for a speaker system that’s designed to adapt itself to a listening room?
Well, it appears that the vaunted CR tests are little better than what an ordinary subjective high-end audio magazine does, despite the pretensions. The listening room, for example, is small with a couch, and no indication of any special setup in terms of carpeting or wall treatment. Or is it meant to represent a typical listening room? Unfortunately, the article isn’t specific enough about such matters.
What is clear is that the speakers, the ones being tested and those used for reference, are placed in the open adjacent to one another. There’s no attempt to isolate the speakers to prevent unwanted reflections or vibrations.
Worse, no attempt is made to perform a blind test, so that a speaker’s brand name, appearance or other factors doesn’t influence a listener’s subjective opinion. For example, a large speaker may seem to sound better than a small one, but not necessarily because of its sonic character. The possibility of prejudice, even unconscious, against one speaker or another, is not considered.
But what about the listening panel? Are there dozens of people taking turns to give the speakers thorough tests? Not quite. The setup involves a chief speaker tester, one Elias Arias, and one other tester. In other words, the panel consists of just two people, a testing duo, supposedly specially trained as skilled listeners in an unspecified manner, with a third brought in in the event of a tie. But no amount of training can compensate for the lack of blind testing.
Wouldn’t it be illuminating if the winning speaker still won if you couldn’t identify it? More likely, the results might be very different. But CR often appears to live in a bubble.
Speakers are measured in a soundproof room (anechoic chamber). The results reveal a speaker’s raw potential, but it doesn’t provide data as to how it behaves in a normal listening room, where reflections will impact the sound that you hear. Experienced audio testers may also perform the same measurements in the actual listening location, so you can see how a real world set of numbers compares to what the listener actually hears.
That comparison with the ones from the anechoic chamber might also provide an indication how the listening area impacts those measurements.
Now none of this means that the HomePod would have seemed less “muddy” if the tests were done blind, or if the systems were isolated from one another to avoid sympathetic vibrations and other side effects. It might have sounded worse, the same, or the results might have been reversed. I also wonder if CR ever bothered to consult with actual loudspeaker designers, such as my old friend Bob Carver, to determine the most accurate testing methods.
It sure seems that CR comes up with peculiar ways to evaluate products. Consider tests of notebook computers, where they run web sites from a server in the default browser with cache off to test battery life. How does that approach possibly represent how people will use these notebooks in the real world?
At least CR claims to stay in touch with manufacturers during the test process, so they can be consulted in the event of a problem. That approach succeeded when a preliminary review of the 2016 MacBook Pro revealed inconsistent battery results. It was strictly the result of that outrageous test process.
So turning off caching in Safari’s usually hidden Develop menu revealed a subtle bug that Apple fixed with a software update. Suddenly a bad review become a very positive review.
Now I am not going to turn this article into a blanket condemnation of Consumer Reports. I hope there will be more details about testing schemes in the next part, so the flaws — and the potential benefits — will be revealed.
In passing, I do hope CR’s lapses are mostly in the tech arena. But I also know that their review of my low-end VW claimed the front bucket seats had poor side bolstering. That turned out to be totally untrue.
CR’s review of the VIZIO M55-E0 “home theater display” mislabeled the names of the setup menu’s features in its recommendations for optimal picture settings. It also claimed that no printed manual was supplied with the set; this is half true. You do receive two Quick Start Guides in multiple languages. In its favor, most of the picture settings actually deliver decent results.
Without including a single player or union rep, the NFL has finally decided on a formal policy in regards to player protests during the National Anthem. NFL owners have decreed that players must stand during the anthem to show “respect;” however, there is an option for players to remain in the locker room.
Also, each team will be able to decide if there should be any discipline for a player protesting on the field during the anthem. Finally, the NFL will be able to fine a team (not a player) if any of the team’s said players protest on the sideline.
So, if a player decides to protest he can do so in the locker room where no one is watching. But if the player decides to come onto the field during the anthem and protests - the NFL fines the team and the team decides if there is any punishment for the player.
I have mixed feelings about the policy. On one hand, the NFL is a company and the players are employees within. And, in a regular business setting the company has the right to determine specific policies that employees must adhere to - dress code, workplace conduct, tardiness policies, etc, etc. Let’s face it, if your boss tells you that you have to wear a tie - and you decide to never wear ties - don’t be surprised when you’re fired. Because, a company should have some rights in deciding how their employees represent the company.
On the other hand, the NFL protesters are, by and large - black folks. And NFL owners are, by and large - white dudes. Which means we now have yet another policy where white dudes tell black folks what is appropriate for them to do and not to do. Which, is kind of tiresome. (Also, racist and gross.) And I'm not even getting into the inherent idea that, a person in a truly free society doesn't need to "respect the flag." Someone with the right of free speech is allowed, legally, to not stand and be forced to "respect the flag." It is literally their constitutional right to chose to stand or not, or to sing the anthem or not, and / or to show "respect" for the flag / anthem - or not. I am using a common sense attachment to the word "respect" but, yes, we could also quibble about "what does respect mean, anyway?"
On the other hand, again, as employees of the NFL, the company might be within their rights to say, "As employees of our company - you must stand and sing the national anthem." On the other hand it's not like the NFL has any competition. So, let's say a player says, "Well, free speech is in the constitution and therefor legal, and if your company policy takes away that right from me - then I quit and I'll take my talents elsewhere!" It's not like they can work at some other professional American football league. Their options are kind of limited.
I've already seen the Facebook rage begin. Even on my feed I have friends railing against the ruling and calling for an NFL boycott. I mean, first there was the ludicrous idea that the NFL was going to impose a 15 yard penalty on any team kneeling for the anthem. Which produced some online rage. Thankfully, it looks like that idea was shot down. But still - the very idea of it is insulting to reasonable thought. And now the idea that players can be fined - for exercising their constitutional rights - is burning some folk's britches. And I totally understand.
On the other hand - players are employees of a company who should have some say in how their employees represent them. Maybe. Right? Or is the policy too far? I honestly don't know. Some folks on Facebook clearly think they have the answer. Maybe they're right!
Or maybe they're overreacting.
Anyway, I hate to be wishy washy on the policy but I have mixed feelings about it. To be clear - I don't have mixed feelings about the obvious racial component. The new NFL policy is absolutely designed to make white Americans feel more comfortable watching the NFL. I mean, let's face it - if there is one thing that scares the F out of many white people, it's black folks exercising their constitutional rights and pushing for social change.
I mean, peacefully protesting by kneeling quietly during the national anthem seems pretty benign, right? This probably shouldn't even be a national issue. But it is. Such is the 'Murica we currently find ourselves in.
To be honest, I prefer the NFL's previous policy of - players are free to exercise their free speech. Or not. It's up to them. And I hate the racial component of this policy but I keep getting stuck on - but the NFL is a company and the company should be allowed some say in how their employees represent - the company. But the rage is building and I honestly don't expect the NFL to keep this policy.
I guess we'll see what happens now.
The NFL Players Association (NFLPA) had this to say about the new policy:
The NFL’s full policy statement:
The 32 member clubs of the National Football League have reaffirmed their strong commitment to work alongside our players to strengthen our communities and advance social justice. The unique platform that we have created is unprecedented in its scope, and will provide extraordinary resources in support of programs to promote positive social change in our communities.
The membership also strongly believes that:
1. All team and league personnel on the field shall stand and show respect for the flag and the Anthem.
2. The Game Operations Manual will be revised to remove the requirement that all players be on the field for the Anthem.
3. Personnel who choose not to stand for the Anthem may stay in the locker room or in a similar location off the field until after the Anthem has been performed.
4. A club will be fined by the League if its personnel are on the field and do not stand and show respect for the flag and the Anthem.
5. Each club may develop its own work rules, consistent with the above principles, regarding its personnel who do not stand and show respect for the flag and the Anthem.
6. The Commissioner will impose appropriate discipline on league personnel who do not stand and show respect for the flag and the Anthem.