|
Balls of lights (3): The Questionable Science of Crop Circles
Grassi et al. reply to the internet rapid reaction by Eltjo Haselhoff on
the JSE article
(July 2006)
by Francesco Grassi, Claudio Cocheo, Paolo Russo
Soon after the publication of our paper
on the Journal of Scientific Exploration, Dr. Eltjo Haselhoff, one of
the authors of the articles we criticized, wrote a short reply text, just a
"rapid reaction", as he defined it. He added:
"I will consider writing a
formal reaction, together with the authors of the other two articles [...]
However, as it has become clear to me that Grassi and coworkers have no or
little interest in an honest and constructive scientific discussion, and too
much time and energy has already been wasted in my opinion, I may decide to
spend my time on more useful activities."
That is exactly what he seems to have
decided, since an year has passed and Haselhoff recently confirmed that
he has not written any formal reaction yet. We were waiting for it before
replying, but now we feel we have waited long enough. This article examines all
of the points Haselhoff made in his reply text. Since a long time has passed,
it may be useful to quote most of Haselhoff's text.
"I was surprised to see that a
comment on earlier publications was not published by the same journal that
presented the original papers, in this case Physiologia Plantarum. This
is unusual for scientific communications, and it is fair to assume that if the
paper by Grassi et al. would have been a relevant comment, the editors of Physiologia
Plantarum would not have rejected it. Although Grassi does provide an
explanation for the fact that Physiologia Plantarum rejected his paper -
which, by the way, is a very uncommon subject of discussion in a scientific
communication - his statements are curious and are currently under
investigation."
We confirm it is a very uncommon
subject, but the reason why we had to include it in our paper is clearly shown
in the previous quote: to avoid "fair assumptions" such as Haselhoff's about
the relevance of our paper. We are also confident that Haselhoff has had enough
time to investigate our statements in depth.
"Basically, Grassi's criticism
on my publication is fourfold: [...]"
This is a serious misrepresentation of
our criticism. Our paper raises many more issues than the four ones mentioned
by Haselhoff. Just to list a few: the Beckhampton formation is not analyzed and
does not fit the model, the assumption about thermal expansion of water is
physically inconsistent, the thermal mechanism could and should have been
tested and was not, the criterion for inclusion of samples in the control group
is not defined, central tufts have been excluded, the assumption about "normal"
node elongation is contradicted by the Nieuwerkerk man-made circle... it would
be pointless to summarize the whole article here. (Many of these problems are
already present in the papers by Levengood and Talbott which Haselhoff's paper
develops from, but nonetheless they do affect Haselhoff's paper just as well,
as pointed out in our paper.) Of course Haselhoff is free
to avoid commenting these points, but should not negate their existence in our
paper, nor hide them behind the word "basically".
Then Haselhoff tries to reject the "four" points he claims we made.
"Before I will address - and
reject - these four claims, one fact needs to be emphasized. An important part
of Grassi's analysis is based on extensive field- and laboratory work performed
by myself."
A significant part of our analysis is a criticism to that
work, so it is just obvious that that part is based on it.
"In the year 2003, Grassi
approached me by means of several very kind and polite e-mails, calling himself
a 'crop circle researcher', and asking me if he could get the raw measurement
data that I had collected from a set of crop circles (Nieuwerkerk, 1996). After
I had sent these to him, no further communications about these data, nor about
my related work has taken place."
Much more than that actually happened
(see later).
"(I will not be offended by
the fact that Grassi neglected to acknowledge me in his paper, which would not
only have been a matter of courtesy, but is also quite common in scientific
communications.)"
We are glad to know that he was not
offended. We avoided acknowledging him, as would have been normal in different
circumstances, because our criticism was not mild. There are cultures, such as
Italian, where acknowledging somebody for having contributed to an analysis that
wipes out his own conclusions would not sound courteous, but ironic, maybe even
offensive. We definitely did not want that.
"I will now briefly comment on
Grassi's four main points of concern:
a. Important aspects in the
presented physical model are omitted
This statement, along with
several other points of concern addressed by Grassi throughout his paper, would
have been appropriate if my paper had been a full length article, presenting
original work. However, my paper clearly was a comment on one of the BLT
papers, and hence, not a self-contained publication of original research. The
issues Grassi raises, including the one mentioned above, were implicitly
addressed in my conclusion, when I stated that the commented paper 'stimulates
further study'. I therefore reject Grassi's criticism, as comments on other
scientific publications need to be focused and concise."
Comments should be focused and concise
because they should be really just comments to other people's work. That is why
they are usually not even peer-reviewed. Haselhoff's "comment" is very unusual:
the first part does comment a BLT paper, showing a couple of errors in the
analysis of data; if only the "comment" ended there, it would be perfect. But
then it goes on with original work: a new hypothesis with a new model (BOL),
new data (Nieuwerkerk), new calculations, new conclusions. Such things should
have never been written as a "comment" in the first place. Comments are a
precious resource for quickly highlighting and correcting defects of scientific
papers; they are not intended as a way to publish short papers quickly,
possibly bypassing the peer review process.
We might have criticized Haselhoff's
improper use of "comments" in our JSE paper. We did not, since we were - and
are - more interested in what he wrote than in how he published it. But science
is science, and there is only one way to make it: correctly. There is just no
place in science for extraordinary claims based on inadequate models,
insufficient data and improper calculations. If Haselhoff thinks he has found
such a place in "comments", he should better think again of what science is.
With regard to that "implicitly addressed in my conclusions", see the next
point.
"b. The findings lack
statistical relevance
At the end of my paper I
concluded that 'much more data would have to be analyzed and thorough
statistical studies will be necessary...' It is therefore curious to see that
Grassi uses my own arguments against me. He simply repeats my own conclusions,
which can never be a critical note, despite the fact that Grassi presents it as
such, and even in a denigrating manner. I do agree with Grassi's statement, but
I reject it as a point of criticism on my work."
No, we did not repeat Haselhoff's
conclusions. When the results of any scientific study lack statistical
relevance, the study's outcome is considered "negative". In other words,
scientifically speaking, "the findings lack statistical relevance" actually
means "there are no findings". Haselhoff never concluded that, of course. On
the contrary, he wrote "The experimental data [...]
suggest that pulvinus length expansion in crop circles is a thermo-mechanic
effect, possibly induced by a kind of electromagnetic point source. Data
obtained from a simple hand-made formation did not reveal the same
characteristics." So Haselhoff claims he did find something: statistical
characteristics, suggesting a physical phenomenon. He also added: "[...] the position-dependent pulvinus length, and in
particular the apparent organized character of the data analysed, is
interesting and stimulates further study." Again, he writes about
characteristics found in the data by his statistical analysis and deserving
further study. A statistical analysis can be said to have found something only
when the result is statistically significant.
Haselhoff's self-quoting from his
conclusions is too short to appreciate the meaning of the original sentence.
The full quote is: "By no means does the author pretend
to present a 'lithmus test' for distinction between a 'genuine' crop formation,
whatever it may be, and a hand-flattened area of crop. Much more data would
have to be analyzed and thorough statistical studies will be necessary before
such a criterion can be defined." So Haselhoff just claims that his
findings are not so universal to be used as a 'genuinity' test for all crop
circles - not yet, at least. Therefore, although Haselhoff's conclusions are
indeed a bit vague and allow some room for interpretations, this room is not
unlimited. He never concluded that he identified no evidence at all for the BOL
hypothesis. He never concluded that he found nothing; nobody ever does. If you
think that you found nothing, you do not write a paper, nor a comment, nor
anything else, unless you do want to make clear that the result of your study
was negative: you looked for something that seems not to exist and you want to
publish this negative evidence. Nowhere in Haselhoff's conclusions can such a
negative admission be found.
Just as an ending note, we are surprised by the striking
contrast between Haselhoff's recent minimization of his own work (just a
comment, just a stimulus for further research, never meant to be anything else)
and the bombastic way he presents it in his book (The Deepening
Complexity of Crop Circles, see
www.deepeningcomplexity.com):
"[...] all linear
regression coefficients had values close to 1, which is a significant proof for
demostrating that a point-like electromagnetic source had caused the swelling
of nodes. [...] These discoveries were presented to a scientific journal [...]
The fact that my article was published is rather important, as it strengthens
the hypothesis that ''balls of light'' are directly involved in the creation of
crop formations (or at least some of them). This is not a mere hypothesis
any more, but an accepted and scientifically proven fact and it will
continue being so until somebody else will offer a different explanation [...]
or will demostrate that the analysis was wrong." (our translation from
the Italian edition, pg. 89-90; text highlighting is our too).
This quote may be useful to clear any
reasonable doubt on what Haselhoff meant in his paper's conclusions, although
he now seems to have changed his mind. We
welcome this evolution of his official position, which we consider a positive
effect of our work, but we reject any redefinition of his previous position:
the past cannot be rewritten. By the way, in that quote Haselhoff considers the
possibility that somebody may demonstrate that his analysis was wrong; he looks
very open to criticism, as any researcher should be. However, we think we did
supply the demostration he was asking for, but now Haselhoff seems rather
unwilling to accept it.
"c. There is a lack of
detailed information and tables with original data should have been provided
This point of criticism is
curious, as I have provided Grassi with all the original data that I had
available."
Curiously, most data we requested have
always been unavailable. We asked Haselhoff for the data of all formations he
analyzed: Devizes, Chehalis, Sussex (sampled by BLT), Nieuwerkerk and Hoeven
(sampled by Haselhoff). Although we had been assured that we would have
received all of them, we received only the Nieuwerkerk data. In all other cases
we had to work with data which had been read back from the graphs.
"Moreover, anyone with just a
little experience in scientific communication knows that publication of tables
with original data is not only unusual, it is even against the guidelines of
basically all scientific journals. Original data are found in log books or
computer spread sheets, and should be available upon request, but they are not
published in scientific communications. This is what I was taught in the first
year of University, and for good reasons: otherwise my paper would have been
twenty pages long in stead of two, and consist mainly of numbers. I therefore
reject this point of criticism too."
We are surprised that Haselhoff did not
understand our point. We did supply three data tables in our paper; all
together, they barely filled a page of the Journal of Scientific Exploration,
and would fill less than half a page in Physiologia Plantarum, whose
pages are denser. Of course the tables just contained the samples averages, not
the single values and even if the averages should be sufficient to reproduce and
verify the regressions, they are absolutely inadequate to provide information
about the uncertainty on the x-axis; this information too should have been
provided in Haselhoff's and BLT's papers. On the contrary, no data table was
present.
Reproducibility can never be an
afterthought in science: it is its key feature. Without tables, anybody willing
to verify the analysis should either get the data from the graphs by hand with
ruler and square, slowly, with limited precision, or ask the author for the
data and wait and hope. Both options are rather slow and uncomfortable and discourage
verification; we would not be surprised if we happened to be first in verifying
Haselhoff's analysis. Not providing data tables is not always a serious fault,
but is seldom a good choice when tables are small; in a paper advancing
extraordinary hypotheses on the basis of just a few samples, it is an
incomprehensible and questionable choice at least.
Moreover, we questioned the lack of
other kinds of information. For example, the shapes of the formations were not
shown.
"d. I deliberately held back
measured data in order to manipulate the statistics
This is more a severe
accusation of fraud rather than a point of criticism. Grassi writes that, after
he received the digital spreadsheet with raw data from me, he 'discovered'
measurements that I had not published, and which, when included in his
analysis, would significantly change the outcome of my findings."
Of course we never accused anybody of
fraud; we just cannot imagine a fraudulent researcher who e-mails the proof of
his deed to somebody requesting it. We thought that was obvious. We just
noticed that something was wrong from the point of view of scientific
methodology (in normal life you may decide not to use all the information you
have at your disposal, in science you have no such freedom), but we had already
noticed other instances of that same kind of error in Haselhoff's work, so we
were ready to accept that final surprise.
"However, the reason for
omitting the data in my analysis was simple: due to an unfortunate incident
several of the samples in this series had been mixed up even before the
measurements were made. Although I had re-ordered the samples to the best of my
knowledge, this event rendered any correlation analysis worthless, including
much of the work performed by Grassi and presented in his paper. All of his
related findings and conclusions are therefore worthless as well.
Interestingly, a simple e-mail would have prevented him and his coworkers from
wasting valuable time."
Interestingly, we did send that e-mail.
Here is a brief summary of what happened.
When we began our enquiry, we not only
read the BLT papers and Haselhoff's paper, but Haselhoff's famous book too.
The paper mentioned the analysis of a man-made circle: "Nieuwerkerk, 1997".
The book mentioned the analysis of a man-made circle: "Dreischor, 1997".
The corresponding graphs were completely different.
However, a little investigation showed
that they were actually the same circle. We suspected that, because Haselhoff
already did something similar with the other formation he examined, named
"Hoeven" in a web page and "Noord-Badant" in his book. We shall not discuss
here the possible reasons for this strange and misleading habit.
At this point we thought there was
something wrong with the graphs. This is one of the reasons why we privately
asked Haselhoff for the Dreischor-Nieuwerkerk data. This is a short excerpt
from Grassi's first e-mail to Haselhoff:
"As per the Nieuwerkerk formation, from
the paper appears that you personally sampled only 6 points, but the values
reported in a graph in your book are different.
Did you sample other points in the Nieuwerkerk formation?
Is it easy for you to send me also the
complete electronic data you have gathered at Nieuwerkerk?"
Haselhoff replied with an e-mail with a
data file attached. The text of the e-mail quoted that excerpt, so it is safe
to assume that Haselhoff read it, but after the quote he just replied to look
at the attached file. So we did.
When we saw that the file contained two
distinct data sets, it became evident that Haselhoff had used set A in the
paper and set B in the book. A closer inspection revealed that he had also
adopted two different values for the control level, although he collected only
one group of control samples; while the paper correctly used the average of all
control samples, the book incorrectly used only part of them. Thanks to this
data selection, the node lengthening plotted in the book's graph is lower than
it should, so leading to an underestimation of the natural node lengthening and
favoring the BOL hypothesis: one of the samples is even attributed a negative
lengthening. That lone sample under the zero level is just one of the features
that make the two graphs easily distinguishable at a first glance.
We just thought we were facing two data
selections, a kind of scientific mistake that we already found in other parts
of Haselhoff's paper (central tufts, Beckhampton) and his Hoeven report (see later).
Nothing new. Why should we have suspected that set B was bad, since Haselhoff
published it in his book?
On the contrary, Haselhoff's explanation
really surprises us. An unfortunate incident may happen, we have no problem
with that; it is the overall chain of unlikely events implied by that
explanation that we find astonishing:
1) An unfortunate incident mixed up the
data, trashing set B.
2) Haselhoff made a mistake and
published the graph of set B in his book. A mistake is the justification he
supplied when asked in the Italian newsgroup it.discussioni.misteri:
http://tinyurl.com/lesod. It
seems he wanted to publish set A in the book, but he happened to publish set B
instead.
3) He made another mistake in the same
graph: he somehow replaced the original control level with a badly recomputed
one.
4) During the whole process of preparing
the graph, placing it in the page, adjusting colors and dimensions, checking
the final print quality, taking care of the many translations and editions of
the book, he never realized that the graph was completely wrong. He never
spotted the sub-zero outlier. Just a cursory glance at the graph would have
probably detected it.
5) After he spent lots of time and
efforts collecting and analyzing data and an incident ruined half of his work,
he just forgot to mention this detail when he sent us the data. This is another
mistake that most people would find very hard to make.
6) We did point at the graph mismatch in
our first e-mail and he quoted it. He did not take any corrective action. He
reacted to the question asked in it.discussioni.ufo as he had never been told
before. It seems he did not take what we wrote - and he read - into any
consideration: yet another mistake.
We leave any judgement on this long chain to the reader.
Apart from the astonishing mix of
misfortune and lack of care that would be necessary to cause all that, it
should be exceedingly clear that if somebody made a mistake it was definitely
not us.
Haselhoff seems to think that this
"mistake" invalidates our paper's conclusions, but anybody accepting his
explanation should notice that his implication is not correct: only a small
part of our work was about the man-made Nieuwerkerk formation, so small in fact
that we did not even care to mention it explicitly in our conclusions. In
particular, our sentence "the 1/r2 symmetry exists only as a
consequence of the unjustified exclusion of unwanted data" mostly refers to
other data exclusions, that Haselhoff has not commented: the central tufts of
the Chehalis and Sussex formations and the entire Beckhampton formation.
"At this point I should add that Grassi's requests for
'more evidence' had already been addressed. This is still the focus of my
personal crop circle activities today. One particular case (Hoeven 1999), had
already been worked out in detail and published in my latest book, which is in
Grassi's possession and formed the basis of his studies."
We confirm we studied the Hoeven case too.
"Grassi also requested the raw
data of this formation, however, I did not have these readily available. Since
then, November 2003, I did not hear from Grassi anymore."
That is a very reductive view.
April 2003:
Grassi sends his first message, qualifying himself as a crop circles
researcher, correctly making clear he is going to verify Haselhoff's claims and
asking for all data analyzed in the paper. He receives just an automatic reply;
Haselhoff is temporarily unreachable.
May 2003:
Grassi tries again. Haselhoff sends the Dreischor data and adds that he will
send the data of Devizes, Chehalis, Sussex and Hoeven too, as soon as he
will be able to retrieve them (!).
November 2003:
Grassi asks again. Haselhoff could not yet retrieve the data (!), but ensures
he will try.
February 2004: one
of us (Russo), who got a particular interest in the Hoeven case, decides to
make one last attempt to get the data. His e-mail makes clear that he is
requesting the data in order to verify Haselhoff's analysis. He receives no
reply.
"However, besides the detailed
information in my book, Grassi was also aware of the extensive report about
this case that had been published long ago (1999) on the internet1,
including raw data2."
1 http://archiv.fgk.org/99/Berichte/Hoeven99/index.shtml
2 http://www.dcccs.org/sample.htm
(currently not on-line any more)
We would have never asked Haselhoff for
the Hoeven data, nor would he have been unable to send them to us, if they had
already been put online. Of course they were not; the Hoeven data consist of
sets A, B, C, D, E, F, G, Co (Haselhoff's Hoeven report1 contains
their graphs, without numerical values); only one raw set was ever put online2
(much later), in a really raw form (images of stems, not measured node
lengths). A single set is valueless for a correct analysis. Haselhoff's web
page does not specify which set he put online, but a quick comparison with the
graphs in the report reveals it is set B.
"This study discusses a case
of a crop circle that fits the BOL hypothesis with very high statistical
relevance, and hence is in conflict with Grassi's main conclusion that "involvement
of some kind of electromagnetic radiation … is not supported by the available
evidence". "
Among all Hoeven data sets, only set B
matches the BOL model quite well and it is evident that this is the reason why
it was chosen for publication online. It is actually deeply incorrect to single
out that set from the data for any statistical analysis: it is just another
data selection error. We verified Haselhoff's analysis of the Hoeven formation;
the results can be found in the article linked here: English,
Italian. Briefly: Haselhoff's
analysis is affected by some errors (mostly data selections); the corrected
results show no reasonable evidence for BOLs.
"One can only speculate why
Grassi decided no to include these data in his analysis."
It would have been the wrong place. The
Hoeven report had never been published in a scientific journal, so there was no
need to criticize it in a scientific journal. We felt that if we did, we might
have implicitly credited the report with a scientific status it did not
deserve. Moreover, our paper was already pretty long. So we published our
analysis of the Hoeven case in the magazine of CICAP, Scienza &
Paranormale (S&P N. 63 - Year XIII
- Sep/Oct 2005), and put it online too.
"Finally, I would like to
express my astonishment about the bombastic and denigrating style of Grassi's
paper, in which he pretends to debunk a sensational claim. Anyone who reads my
paper will agree that this was a mere comment to the work of the BLT team,
suggesting some model adaptations and carrying ahead their hypotheses with a
modified version, only to stimulate further study."
We have already commented Haselhoff's
recent just-a-comment position, but an additional remark may be useful here.
Haselhoff seems to think that we should not have criticized his paper because
it is "a mere comment". Scientifically
speaking, there is a precise difference between a peer-reviewed paper and a
simple comment. A comment is the expression of the opinion (opinion, not
science) of somebody about the work carried out by somebody else, with the
bonds imposed by the editorial board of the review only (and no checking by the
scientific board). Moreover, while a comment should enjoy the advantage of
being not scientifically criticizable, as an expression of the free thinking of
someone, at the same time it should not pretend to scientifically prove
anything but a subjective personal thinking.
We already remarked that Haselhoff's
"comment" is not a mere comment at all, since it contains new
hypotheses, data, calculations and conclusions, just like a full paper, and so
it is also susceptible of technical criticisms, as we did. But even if we
accepted - and we do not - Haselhoff's opinion that his work should not be
commented at all because it is - formally - not a full paper, he would run into
another problem. We have just criticized Haselhoff's claim that his Hoeven
report supplies the scientific evidence that was missing from his paper, but
apart from our criticism, the Hoeven report has never been published in any
scientific journal, not even as a comment. Coherently with his recent view that
nothing less than a formal peer-reviewed full paper should ever be considered
as an attempt to scientifically prove anything, he should have never mentioned
his Hoeven report as a source of scientific evidence in the first place.
"In my opinion the style
of Grassi's comment, as well as the propaganda related to it that he currently
carries out over the internet and beyond is way out of proportion, and casts a
dark shadow over his true intentions."
Propaganda? It is not us who perform
data selections. We analyze available information and publish the results. We
are interested only in claims on the paranormal and in facts that should back them.
Therefore, we will not comment Haselhoff's style nor draw any conclusion about
his true intentions (although readers may do): that is definitely beyond our
interests.
Haselhoff's "comment" paper highlights a
couple of very serious errors in the BLT paper it refers too, actually invalidating
its conclusions. This is a serious criticism, however presented. Any researcher
has the scientific right to criticize anybody else's work. Any serious
researcher knows that rule; science would never work otherwise. The primary
reason why a scientist must be so careful on his work is that he knows all too
well that if he makes mistakes, somebody else will find them. Haselhoff was
just in his right when he pointed out BLT's errors; so were we.
Moreover, we ourselves directly invited Haselhoff, and Levengood & Talbott on behalf of
Haselhoff, to write an official reply and to submit it to JSE journal.
We were not satisfied by the "rapid
reaction" published by Haselhoff on the DCCCS website, because it was not a
true scientific criticism but a simple opinion. We waited till now, expecting a
scientific publication from both Haselhoff and Levengood & Talbott that
deeply and with well founded reasons criticizes our article on the JSE.
We are not closed to any scientific
criticism on our work, that obviously is at disposal of anyone who would like
to investigate it. So far we have had no reply.
"Conclusion
Normally, authors of
scientific communications are asked for comment before their work is
criticized. This is not just a matter of courtesy, but it also avoids precious
time being wasted. Any serious researcher would take the time to verify if
there is some fact they have overlooked or possibly misunderstood before they
rush to publications and press releases. I can only conclude that Mr Grassi has
little or no experience in scientific communication. Open and honest
correspondence, initiated by Mr Grassi and co-authors would have saved myself,
them, as well as the editors of Physiologia Plantarum and the Journal of
Scientific Exploration much unnecessary work. I do appreciate efforts that are
dedicated to the pursuit of knowledge. However, I can't help suspecting from
the course of matters that Mr Grassi was more dedicated towards disgracing me
and my work rather than to performing honest and sound research."
Haselhoff's point would have some value
if our paper had really been just a waste of time, i.e. if all points made in
it had turned out to be groundless. On the contrary, all of our conclusions
still hold. About the Nieuwerkerk data: not even the incident hypothesis would
turn our work on this single point into a waste of time. At least, it would
have lead to the identification of an error whose existence had never been
acknowledged before, i.e. that the data set published in Haselhoff's book was
completely wrong.
We did take the time to verify our
analysis as deeply as possible. There is a reason why our paper has three
authors: we discussed every single point to great lengths. We also asked
several other experts for their opinions. This process took a lot of time, but
it was well spent. The only waste of time we met during this study was the
hopeless wait for most of the data we asked Haselhoff for. However, on two
occasions (Grassi, April 2003; Russo, February 2004) we did try to discuss
obscure points of his work with him, with no results. We just gave up.
Francesco Grassi
Ingegnere (Dr. Eng.), CICAP
Experimentation Group
Claudio Cocheo
Centro di Ricerche Ambientali,
Fondazione Salvatore Maugeri, CICAP Veneto
Paolo Russo
Programmer, CICAP Friuli Venezia
Giulia
They all are members of the CICAP Study Team on Crop
Circles
|
