Tuesday, January 31, 2012

CASARA, Electronic Searches - Science vs Pareidolia

This is the fourth article in a series that starts with CASARA, Arctic First Response and Electronic Searches.

The scientific method puts the burden of proof on the person, or group, that advances a theory. This puts the onus on Barr, Casey and Muir to prove their technique is effective by providing evidence or through mathematical proofs. Langley Muir holds a PhD so he should be fully aware of the scientific method. Anne Barr is listed as an engineering member of the Carleton University Advanced Cognitive Engineering Lab. The VSIM ACE Lab mission "is to discover fundamental principles of human perception and cognition and to apply these principles to the design, implementation and evaluation of advanced human-machine systems." So I would have expected she at least would be familiar with the topics I'm going to discuss in this article. The VSIM ACE Lab lists CASARA as a partner and collaborating organization, so she should have been able to draw on the advice of the scientist there to plan a testing procedure. Even though I've lowered the bar by asking them simply to provide a scientific rebuttal to the issues and concerns I've raised about their techniques, they are either unable or unwilling to do even that.

Since the president of CASARA Ontario considers this "two very knowledgeable people at odds regarding the Theory of the Aural Null Procedure", why don't we see what we can infer about how knowledgeable the other person (whom ever that may be) is regarding the theories and science applicable to the Aural Null Procedure.

Bar, Casey and Muir use the records from one flight to demonstrate their procedures published in a paper (which I introduced in my first article). This is the very same flight, documented by Langley Muir, that is the basis for my paper as well. I have plotted (as best I can without the raw data) the reported and actual ELT locations in the search area used for that flight on Google Maps. In their paper where they describe this flight they claim "the location was chosen so that the radiation pattern of the ELT would not be distorted from circular by any reflectors or obstructions." One can easily see many potential reflectors in the Google satellite photo. The training ELT is placed near a parking lot, which is in turn surrounded by agricultural fields, both of which often make good reflectors. By selecting the map view and adding the terrain layer one can see the ground slopes away East-North-East to a broad flat stretch of the Gatineau River which could make a fine reflector indeed. There are also numerous bluffs and cliffs surrounding the location, which may or may not (depending on shape, structure, overburden, etc) be good reflectors, but will certainly be obstructions. This also means that they knew the location of the ELT prior to the flight.

You may recall the description of the Cardinal Pass from my last article in the series. As a quick reminder, the search airplane flies past the rough location of the ELT with the crew listening to the sound of the ELT on the aircraft radio, off-tuned to reduce sensitivity. The hypothesis is that at the closest point of approach the strength of the signal from the ELT will be strongest resulting in the audio output of the demodulated ELT signal also being strongest at that point. This flight was recorded using a handheld video recorder. The audio track was recorded from a lapel microphone inserted into the ear cup of one of the crew's headsets. As unit training officer I was provided a copy of the video on DVD along with the flight documentation. I am a signals analyst with 33 years of experience so I was able to extract the audio track and perform spectrographic analysis. The distinctive sweeping of the ELT aural signal made it easy to identify and isolate on the recording. From that point it was a simple matter to measure the amplitude of the signal presented to the crew. As the aircraft neared the ELT and the signal reached the sensitivity limit of the receiver the ELT signal may be observed in the spectrogram. The ELT signal increases in amplitude, except for a small number of fades suggestive of multi-path, until the receiver AGC minimum amplitude is reached. The ELT audio signal amplitude then remains constant until the aircraft reaches the other end of the pass or otherwise moves away from the ELT and the signal amplitude drops below the receiver AGC operating region. The amplitude then fades out and in again in response to multi-path, until the signal is below the receiver maximum sensitivity and the ELT can no longer be detected. There is a change in the quality of signal "tone" throughout the pass. This is likely attributable to a shift in relative phase between the carrier and the sidebands caused by the multi-path propagation of the signal. Phase shifts of this nature cause distortion of the demodulated signal. The signal amplitude is constant for several seconds before and after the navigator indicated peak signal passage. No detectable peak ELT signal was found. So why are they sure they can hear an amplitude peak of the signal? The best explanation I can come up with is pareidolia. The combination of the fact that the crew knew where the ELT was before they started the search, which provides a strong subconscious bias, with the vague aural data, perhaps the changing tone, and a desire to demonstrate the effectiveness of the technique is all that is necessary for the brain to take over and fill in the gaps.

Training ELT no off-tuning 123.1MHz
The image to the left shows the audio spectra FFT (top) and waterfall display (bottom) of the training beacon with the receiver tuned to the beacon frequency 123.1 MHz (no off-tuning). The start and stop of sweep frequencies can be clearly seen, as can the second harmonic sweeping in time with the fundamental.





Training ELT off-tuned 123.15MHz
This image shows the audio spectra of the training beacon during one of the cardinal passes, off tuned to 123.15 MHz. Note that the start and stop frequencies are at the first harmonic of where they are when the receiver is tuned to the beacon frequency. This would be caused by the same mechanism that results in the spurious transmission appearing on 123.15 MHz. This is one more reason why crews can not depend on the transmitter, receiver or signal behaving the same way or producing a signal similar to the designed transmission on (in this case) 123.1 MHz. The amplitude roll off towards the top of the spectra is consistent with the frequency response of the receiver. I have posted some videos of the spectral analysis running in an addendum to this article.

UPDATE: I was sent a request to try to come up with a simpler display of the volume vs time for the Cardinal Pass. This posed some unique challenges so it took some time to develop a working technique because we are interested in the volume of a signal that is continuously changing frequency and there are other signals (the crew talking, other aircraft communicating). I finally came up with three plots of the volume over time in three frequency bands: 1400 Hz, 1700 Hz and 2000 Hz:


The navigator of the search aircraft called peak volume and marked the aircraft abeam the ELT at time index 9.92 though you can clearly see that in each of the three graphs there are higher peaks at other times. Of course these graphs have been enhanced to make changes of volume easier to detect. The human ear works on a logarithmic scale. The following graph are the same data plotted on a logarithmic scale to give you an idea of what the ear whould hear. The four traces are the same frequencies from above, pluse the full ELT tone sweep spectrum (and times have been reformatted to minues:seconds):




Not much to work with is it? Is the aircraft abeam the ELT at 09:55 as the navigator reported? Or is the real peak at 10:05? No serious signals analyst, engineer or scientist would ever trust the human ear to make a measurement like this, especially when health and safety is involved. No serious signals analyst, engineer or scientist would use a receiver with a fixed autmatic gain control as the sensor to make a measurement like this. Human hearing is simply too easy to fool, there are too many individual characteristics of hearing acuity. Most of the processing is performed by the perception centres of the brain. Conscious decisions based on audio stimulation are far too easily influenced by personal bias as Michael Shermer demonstrates if the following video.

Michael Shermer demonstrated Audio Pareidolia in this clip from TED 2006:

(Many more examples of "messages" in music played backwards are available from this website.) Michael talks about Pareidolia and self deception in general at TED 2010. And if you are interested he gave an in depth talk and answered questions at a book reading in New York June 2011. During TED 2010 Michael shows us a device sold to the Iraqi government, the ADE 651 built by ATSC Ltd. There is more information available on ATSC and the ADE 651 in this BBC Newsnight report:

Update: The video has been pulled from YouTube but is still available from the BBC directly at this address: http://news.bbc.co.uk/2/hi/programmes/newsnight/8471187.stm

This video should be required viewing for any manager before accepting claims in support of a device or technique without peer reviewed scientific data. Of special importance is the effort to which the Iraqi security forces go to defend the device once put into service. This is evident in the press conference footage at 4:32 into the video. This link will take you directly to that time index.

If we turn back to Mike Casey's email in my previous article, you can see phraseology that also indicates subconscious bias at work on that flight:
  • "Bets on the on the flight at the time were on a structure/barn on the Quebec side of the river.";
  • "and with prior knowledge of Hawkesbury East PG5 (453507433), the focus moved from the structure to the airport."
There is also language indicative of the perception that the amplitude of the signal can be heard despite the action of the receiver AGC:
  • "As the flight flew over the town the off tuning was more pronounced with a rapid drop in signal strength as the flight continued north of the river."
  • "The audible signal did increase but began to decrease as the flight approached the bridge."
  • "The flight continued past the bridge with the audible signal rapidly decreasing effectively eliminating NV4 as the source."
  • "The flight continued on its heading with the audible signal increasing after passing the structure."
  • "With greater and greater off-tuning the audible signal increased as the flight entered Ontario. When the audible signal began to decrease, ..."
  • "resulting in a rapid increase in audible signal peaking as the flight passed over the airport and an equally rapid decrease in audible signal as the flight continued along its heading."
Unfortunately there was, to my knowledge, no data collected on that flight to determine which of those statements accurately reflect a change in signal amplitude and which resulted from pareidolia. We are given no reason to suspect that this flight would be any different from the recorded flight. There is a strong subconscious bias first for the structure/barn, then for the airport. It is quite common for accidentally activated ELTs to be found at airports in their airplanes, so the bias, and the success of finding the ELT there is neither surprising nor remarkable.

Clearly the core hypotheses of the Cardinal Pass and the off-tuning augmentation to the Aural Null proposed by Barr, Casey and Muir are disproved; in large part by their own data.

So how can or how should an organization avoid these kind of mistakes? The scientific process is the one saviour we have. The ideal would be mathematical proof, or inductive reasoning based on the mathematical proofs of others in the field. The next would be a statistically significant, double-blind study. It may seem obvious, but first and foremost ensure that the data collected during experiments supports rather than refutes the hypotheses. When the data refutes the hypotheses, it is likely because the hypothesis is not valid.

In fact at one point Muir acknowledged that more testing may be needed; but the form that the testing he suggested was not double-blind, not even single-blind:
I would suggest that we need to more fully test out the method in addition to what is already here.  We should place a target in a fixed location, then do low level N-S passes over the target with the ELT antenna vertical, then horizontal and orientated E-W, then horizontal and orientated N-S, then NE-SW. and finally repeat at 45 degrees elevation. Since the antenna is quite short compared to the aircraft altitude, I would suspect that the null would only be heard when the antenna is vertical and the rest of the orientations would only see a higher gain since the radiation pattern should be symmetrical. We could also experiment with different ground planes to change the antenna gain and/or reflectors to distort the radiation pattern. Having a person on the ground to manipulate the antenna would make a quite large number of passes possible in a very short time. Of course, much of this is pretty well-known antenna theory, but I don't know all that much about the practical aspects; maybe all this messing about is unnecessary.
I believe it is significant that between the time he made this statement, and he collaborated with Barr and Casey on their paper, there was no more data collected to bolster their hypotheses. He is correct in that all of this is pretty well-known antenna theory. If one does not want to do the math to solve for the general case, the math proves intractable or just to avoid all the messing about one can run simulations. Simulation can test far more scenarios than one could hope to practically test in real life; as long as the simulation embodies sufficient theoretical and practical knowledge to provide enough fidelity. Here are two simulations of the Cardinal Pass which assume an airplane equipped with the means to directly measure received signal strength. The magenta lines represent the North-South and East-West passes. The green lines are 90° to the passes at the point of highest signal strength.

Monopole vertical, Cardinals 1.5km Error 0km
Monopole 60° from vertical, Cardinals 3km Error 2.6km
My second article, and my paper contain images from simulations of multi-path signals.

What about a statistically significant sample? There is only the one flight that I know of where any data was collected at all. But Muir claims "both Mike and I have tested the methods out at least a couple of times each, so there may be some merit to all this." So I'm guessing no statistically significant sample either; at least none has ever been provided to me. So no double-blind studies, no statistically significant sample, the math doesn't support their claims, simulation shows that antenna orientation and multi-path can cause errors even when signal strength can be measured directly and their data disproves their hypotheses. This would all be sadly amusing if it were not for the direct impact this could have on the health and safety of the flying public in Canada.

4 comments:

  1. Dude, you've got to be kiddng me! I learned more science in high scool. How do people think they can do that kind of thing. Props for coming forward.

    ReplyDelete
  2. Ditto. I even asked CASARA about this and I was ignored. I was thinking of joining but I think I will have to find something else to occupy my time.

    ReplyDelete
    Replies
    1. That's too bad. Most CASARA members are good people who just want to do a job. It's always the few who spoil it for the many.

      Delete