Comitato Italiano per il Controllo
delle Affermazioni sul Paranormale

Il CICAP promuove un'indagine scientifica
e critica nei confronti del paranormale.
 
 

SPECIAL CROP CIRCLES

Articles

 

Cicap Extra

CICAP

Rubriche

Notizie

Rivista

Enciclopedia

Articoli

Indagini

Scuola

Video

BookShop

Liste

Links

Uff. Stampa

 

Balls of lights (3): The Questionable Science of Crop Circles

Grassi et al. reply to the internet rapid reaction by Eltjo Haselhoff on the JSE article
(July 2006)

by Francesco Grassi, Claudio Cocheo, Paolo Russo


Soon after the publication of our paper on the Journal of Scientific Exploration, Dr. Eltjo Haselhoff, one of the authors of the articles we criticized, wrote a short reply text, just a "rapid reaction", as he defined it. He added:

"I will consider writing a formal reaction, together with the authors of the other two articles [...] However, as it has become clear to me that Grassi and coworkers have no or little interest in an honest and constructive scientific discussion, and too much time and energy has already been wasted in my opinion, I may decide to spend my time on more useful activities."

That is exactly what he seems to have decided, since an year has passed and Haselhoff recently confirmed that he has not written any formal reaction yet. We were waiting for it before replying, but now we feel we have waited long enough. This article examines all of the points Haselhoff made in his reply text. Since a long time has passed, it may be useful to quote most of Haselhoff's text.

"I was surprised to see that a comment on earlier publications was not published by the same journal that presented the original papers, in this case Physiologia Plantarum. This is unusual for scientific communications, and it is fair to assume that if the paper by Grassi et al. would have been a relevant comment, the editors of Physiologia Plantarum would not have rejected it. Although Grassi does provide an explanation for the fact that Physiologia Plantarum rejected his paper - which, by the way, is a very uncommon subject of discussion in a scientific communication - his statements are curious and are currently under investigation."

We confirm it is a very uncommon subject, but the reason why we had to include it in our paper is clearly shown in the previous quote: to avoid "fair assumptions" such as Haselhoff's about the relevance of our paper. We are also confident that Haselhoff has had enough time to investigate our statements in depth.

"Basically, Grassi's criticism on my publication is fourfold: [...]"

This is a serious misrepresentation of our criticism. Our paper raises many more issues than the four ones mentioned by Haselhoff. Just to list a few: the Beckhampton formation is not analyzed and does not fit the model, the assumption about thermal expansion of water is physically inconsistent, the thermal mechanism could and should have been tested and was not, the criterion for inclusion of samples in the control group is not defined, central tufts have been excluded, the assumption about "normal" node elongation is contradicted by the Nieuwerkerk man-made circle... it would be pointless to summarize the whole article here. (Many of these problems are already present in the papers by Levengood and Talbott which Haselhoff's paper develops from, but nonetheless they do affect Haselhoff's paper just as well, as pointed out in our paper.) Of course Haselhoff is free to avoid commenting these points, but should not negate their existence in our paper, nor hide them behind the word "basically".

Then Haselhoff tries to reject the "four" points he claims we made.

"Before I will address - and reject - these four claims, one fact needs to be emphasized. An important part of Grassi's analysis is based on extensive field- and laboratory work performed by myself."

A significant part of our analysis is a criticism to that work, so it is just obvious that that part is based on it.

"In the year 2003, Grassi approached me by means of several very kind and polite e-mails, calling himself a 'crop circle researcher', and asking me if he could get the raw measurement data that I had collected from a set of crop circles (Nieuwerkerk, 1996). After I had sent these to him, no further communications about these data, nor about my related work has taken place."

Much more than that actually happened (see later).

"(I will not be offended by the fact that Grassi neglected to acknowledge me in his paper, which would not only have been a matter of courtesy, but is also quite common in scientific communications.)"

We are glad to know that he was not offended. We avoided acknowledging him, as would have been normal in different circumstances, because our criticism was not mild. There are cultures, such as Italian, where acknowledging somebody for having contributed to an analysis that wipes out his own conclusions would not sound courteous, but ironic, maybe even offensive. We definitely did not want that.

"I will now briefly comment on Grassi's four main points of concern:

a. Important aspects in the presented physical model are omitted

This statement, along with several other points of concern addressed by Grassi throughout his paper, would have been appropriate if my paper had been a full length article, presenting original work. However, my paper clearly was a comment on one of the BLT papers, and hence, not a self-contained publication of original research. The issues Grassi raises, including the one mentioned above, were implicitly addressed in my conclusion, when I stated that the commented paper 'stimulates further study'. I therefore reject Grassi's criticism, as comments on other scientific publications need to be focused and concise."

Comments should be focused and concise because they should be really just comments to other people's work. That is why they are usually not even peer-reviewed. Haselhoff's "comment" is very unusual: the first part does comment a BLT paper, showing a couple of errors in the analysis of data; if only the "comment" ended there, it would be perfect. But then it goes on with original work: a new hypothesis with a new model (BOL), new data (Nieuwerkerk), new calculations, new conclusions. Such things should have never been written as a "comment" in the first place. Comments are a precious resource for quickly highlighting and correcting defects of scientific papers; they are not intended as a way to publish short papers quickly, possibly bypassing the peer review process.

We might have criticized Haselhoff's improper use of "comments" in our JSE paper. We did not, since we were - and are - more interested in what he wrote than in how he published it. But science is science, and there is only one way to make it: correctly. There is just no place in science for extraordinary claims based on inadequate models, insufficient data and improper calculations. If Haselhoff thinks he has found such a place in "comments", he should better think again of what science is.

With regard to that "implicitly addressed in my conclusions", see the next point.

"b. The findings lack statistical relevance

At the end of my paper I concluded that 'much more data would have to be analyzed and thorough statistical studies will be necessary...' It is therefore curious to see that Grassi uses my own arguments against me. He simply repeats my own conclusions, which can never be a critical note, despite the fact that Grassi presents it as such, and even in a denigrating manner. I do agree with Grassi's statement, but I reject it as a point of criticism on my work."

No, we did not repeat Haselhoff's conclusions. When the results of any scientific study lack statistical relevance, the study's outcome is considered "negative". In other words, scientifically speaking, "the findings lack statistical relevance" actually means "there are no findings". Haselhoff never concluded that, of course. On the contrary, he wrote "The experimental data [...] suggest that pulvinus length expansion in crop circles is a thermo-mechanic effect, possibly induced by a kind of electromagnetic point source. Data obtained from a simple hand-made formation did not reveal the same characteristics." So Haselhoff claims he did find something: statistical characteristics, suggesting a physical phenomenon. He also added: "[...] the position-dependent pulvinus length, and in particular the apparent organized character of the data analysed, is interesting and stimulates further study." Again, he writes about characteristics found in the data by his statistical analysis and deserving further study. A statistical analysis can be said to have found something only when the result is statistically significant.

Haselhoff's self-quoting from his conclusions is too short to appreciate the meaning of the original sentence. The full quote is: "By no means does the author pretend to present a 'lithmus test' for distinction between a 'genuine' crop formation, whatever it may be, and a hand-flattened area of crop. Much more data would have to be analyzed and thorough statistical studies will be necessary before such a criterion can be defined." So Haselhoff just claims that his findings are not so universal to be used as a 'genuinity' test for all crop circles - not yet, at least. Therefore, although Haselhoff's conclusions are indeed a bit vague and allow some room for interpretations, this room is not unlimited. He never concluded that he identified no evidence at all for the BOL hypothesis. He never concluded that he found nothing; nobody ever does. If you think that you found nothing, you do not write a paper, nor a comment, nor anything else, unless you do want to make clear that the result of your study was negative: you looked for something that seems not to exist and you want to publish this negative evidence. Nowhere in Haselhoff's conclusions can such a negative admission be found.

Just as an ending note, we are surprised by the striking contrast between Haselhoff's recent minimization of his own work (just a comment, just a stimulus for further research, never meant to be anything else) and the bombastic way he presents it in his book (The Deepening Complexity of Crop Circles, see www.deepeningcomplexity.com): "[...] all linear regression coefficients had values close to 1, which is a significant proof for demostrating that a point-like electromagnetic source had caused the swelling of nodes. [...] These discoveries were presented to a scientific journal [...] The fact that my article was published is rather important, as it strengthens the hypothesis that ''balls of light'' are directly involved in the creation of crop formations (or at least some of them). This is not a mere hypothesis any more, but an accepted and scientifically proven fact and it will continue being so until somebody else will offer a different explanation [...] or will demostrate that the analysis was wrong." (our translation from the Italian edition, pg. 89-90; text highlighting is our too).

This quote may be useful to clear any reasonable doubt on what Haselhoff meant in his paper's conclusions, although he now seems to have changed his mind. We welcome this evolution of his official position, which we consider a positive effect of our work, but we reject any redefinition of his previous position: the past cannot be rewritten. By the way, in that quote Haselhoff considers the possibility that somebody may demonstrate that his analysis was wrong; he looks very open to criticism, as any researcher should be. However, we think we did supply the demostration he was asking for, but now Haselhoff seems rather unwilling to accept it.

"c. There is a lack of detailed information and tables with original data should have been provided

This point of criticism is curious, as I have provided Grassi with all the original data that I had available."

Curiously, most data we requested have always been unavailable. We asked Haselhoff for the data of all formations he analyzed: Devizes, Chehalis, Sussex (sampled by BLT), Nieuwerkerk and Hoeven (sampled by Haselhoff). Although we had been assured that we would have received all of them, we received only the Nieuwerkerk data. In all other cases we had to work with data which had been read back from the graphs.

"Moreover, anyone with just a little experience in scientific communication knows that publication of tables with original data is not only unusual, it is even against the guidelines of basically all scientific journals. Original data are found in log books or computer spread sheets, and should be available upon request, but they are not published in scientific communications. This is what I was taught in the first year of University, and for good reasons: otherwise my paper would have been twenty pages long in stead of two, and consist mainly of numbers. I therefore reject this point of criticism too."

We are surprised that Haselhoff did not understand our point. We did supply three data tables in our paper; all together, they barely filled a page of the Journal of Scientific Exploration, and would fill less than half a page in Physiologia Plantarum, whose pages are denser. Of course the tables just contained the samples averages, not the single values and even if the averages should be sufficient to reproduce and verify the regressions, they are absolutely inadequate to provide information about the uncertainty on the x-axis; this information too should have been provided in Haselhoff's and BLT's papers. On the contrary, no data table was present.

Reproducibility can never be an afterthought in science: it is its key feature. Without tables, anybody willing to verify the analysis should either get the data from the graphs by hand with ruler and square, slowly, with limited precision, or ask the author for the data and wait and hope. Both options are rather slow and uncomfortable and discourage verification; we would not be surprised if we happened to be first in verifying Haselhoff's analysis. Not providing data tables is not always a serious fault, but is seldom a good choice when tables are small; in a paper advancing extraordinary hypotheses on the basis of just a few samples, it is an incomprehensible and questionable choice at least.

Moreover, we questioned the lack of other kinds of information. For example, the shapes of the formations were not shown.

"d. I deliberately held back measured data in order to manipulate the statistics

This is more a severe accusation of fraud rather than a point of criticism. Grassi writes that, after he received the digital spreadsheet with raw data from me, he 'discovered' measurements that I had not published, and which, when included in his analysis, would significantly change the outcome of my findings."

Of course we never accused anybody of fraud; we just cannot imagine a fraudulent researcher who e-mails the proof of his deed to somebody requesting it. We thought that was obvious. We just noticed that something was wrong from the point of view of scientific methodology (in normal life you may decide not to use all the information you have at your disposal, in science you have no such freedom), but we had already noticed other instances of that same kind of error in Haselhoff's work, so we were ready to accept that final surprise.

"However, the reason for omitting the data in my analysis was simple: due to an unfortunate incident several of the samples in this series had been mixed up even before the measurements were made. Although I had re-ordered the samples to the best of my knowledge, this event rendered any correlation analysis worthless, including much of the work performed by Grassi and presented in his paper. All of his related findings and conclusions are therefore worthless as well. Interestingly, a simple e-mail would have prevented him and his coworkers from wasting valuable time."

Interestingly, we did send that e-mail. Here is a brief summary of what happened.

When we began our enquiry, we not only read the BLT papers and Haselhoff's paper, but Haselhoff's famous book too.
The paper mentioned the analysis of a man-made circle: "Nieuwerkerk, 1997".
The book mentioned the analysis of a man-made circle: "Dreischor, 1997".
The corresponding graphs were completely different.
However, a little investigation showed that they were actually the same circle. We suspected that, because Haselhoff already did something similar with the other formation he examined, named "Hoeven" in a web page and "Noord-Badant" in his book. We shall not discuss here the possible reasons for this strange and misleading habit.

At this point we thought there was something wrong with the graphs. This is one of the reasons why we privately asked Haselhoff for the Dreischor-Nieuwerkerk data. This is a short excerpt from Grassi's first e-mail to Haselhoff:

"As per the Nieuwerkerk formation, from the paper appears that you personally sampled only 6 points, but the values reported in a graph in your book are different.
Did you sample other points in the Nieuwerkerk formation?
Is it easy for you to send me also the complete electronic data you have gathered at Nieuwerkerk?"

Haselhoff replied with an e-mail with a data file attached. The text of the e-mail quoted that excerpt, so it is safe to assume that Haselhoff read it, but after the quote he just replied to look at the attached file. So we did.

When we saw that the file contained two distinct data sets, it became evident that Haselhoff had used set A in the paper and set B in the book. A closer inspection revealed that he had also adopted two different values for the control level, although he collected only one group of control samples; while the paper correctly used the average of all control samples, the book incorrectly used only part of them. Thanks to this data selection, the node lengthening plotted in the book's graph is lower than it should, so leading to an underestimation of the natural node lengthening and favoring the BOL hypothesis: one of the samples is even attributed a negative lengthening. That lone sample under the zero level is just one of the features that make the two graphs easily distinguishable at a first glance.

We just thought we were facing two data selections, a kind of scientific mistake that we already found in other parts of Haselhoff's paper (central tufts, Beckhampton) and his Hoeven report (see later). Nothing new. Why should we have suspected that set B was bad, since Haselhoff published it in his book?

On the contrary, Haselhoff's explanation really surprises us. An unfortunate incident may happen, we have no problem with that; it is the overall chain of unlikely events implied by that explanation that we find astonishing:

1) An unfortunate incident mixed up the data, trashing set B.

2) Haselhoff made a mistake and published the graph of set B in his book. A mistake is the justification he supplied when asked in the Italian newsgroup it.discussioni.misteri: http://tinyurl.com/lesod. It seems he wanted to publish set A in the book, but he happened to publish set B instead.

3) He made another mistake in the same graph: he somehow replaced the original control level with a badly recomputed one.

4) During the whole process of preparing the graph, placing it in the page, adjusting colors and dimensions, checking the final print quality, taking care of the many translations and editions of the book, he never realized that the graph was completely wrong. He never spotted the sub-zero outlier. Just a cursory glance at the graph would have probably detected it.

5) After he spent lots of time and efforts collecting and analyzing data and an incident ruined half of his work, he just forgot to mention this detail when he sent us the data. This is another mistake that most people would find very hard to make.

6) We did point at the graph mismatch in our first e-mail and he quoted it. He did not take any corrective action. He reacted to the question asked in it.discussioni.ufo as he had never been told before. It seems he did not take what we wrote - and he read - into any consideration: yet another mistake.

We leave any judgement on this long chain to the reader.
Apart from the astonishing mix of misfortune and lack of care that would be necessary to cause all that, it should be exceedingly clear that if somebody made a mistake it was definitely not us.

Haselhoff seems to think that this "mistake" invalidates our paper's conclusions, but anybody accepting his explanation should notice that his implication is not correct: only a small part of our work was about the man-made Nieuwerkerk formation, so small in fact that we did not even care to mention it explicitly in our conclusions. In particular, our sentence "the 1/r2 symmetry exists only as a consequence of the unjustified exclusion of unwanted data" mostly refers to other data exclusions, that Haselhoff has not commented: the central tufts of the Chehalis and Sussex formations and the entire Beckhampton formation.

"At this point I should add that Grassi's requests for 'more evidence' had already been addressed. This is still the focus of my personal crop circle activities today. One particular case (Hoeven 1999), had already been worked out in detail and published in my latest book, which is in Grassi's possession and formed the basis of his studies."

We confirm we studied the Hoeven case too.

"Grassi also requested the raw data of this formation, however, I did not have these readily available. Since then, November 2003, I did not hear from Grassi anymore."

That is a very reductive view.

April 2003: Grassi sends his first message, qualifying himself as a crop circles researcher, correctly making clear he is going to verify Haselhoff's claims and asking for all data analyzed in the paper. He receives just an automatic reply; Haselhoff is temporarily unreachable.

May 2003: Grassi tries again. Haselhoff sends the Dreischor data and adds that he will send the data of Devizes, Chehalis, Sussex and Hoeven too, as soon as he will be able to retrieve them (!).

November 2003: Grassi asks again. Haselhoff could not yet retrieve the data (!), but ensures he will try.

February 2004: one of us (Russo), who got a particular interest in the Hoeven case, decides to make one last attempt to get the data. His e-mail makes clear that he is requesting the data in order to verify Haselhoff's analysis. He receives no reply.

"However, besides the detailed information in my book, Grassi was also aware of the extensive report about this case that had been published long ago (1999) on the internet1, including raw data2."

1 http://archiv.fgk.org/99/Berichte/Hoeven99/index.shtml

2 http://www.dcccs.org/sample.htm (currently not on-line any more)

We would have never asked Haselhoff for the Hoeven data, nor would he have been unable to send them to us, if they had already been put online. Of course they were not; the Hoeven data consist of sets A, B, C, D, E, F, G, Co (Haselhoff's Hoeven report1 contains their graphs, without numerical values); only one raw set was ever put online2 (much later), in a really raw form (images of stems, not measured node lengths). A single set is valueless for a correct analysis. Haselhoff's web page does not specify which set he put online, but a quick comparison with the graphs in the report reveals it is set B.

"This study discusses a case of a crop circle that fits the BOL hypothesis with very high statistical relevance, and hence is in conflict with Grassi's main conclusion that "involvement of some kind of electromagnetic radiation is not supported by the available evidence". "

Among all Hoeven data sets, only set B matches the BOL model quite well and it is evident that this is the reason why it was chosen for publication online. It is actually deeply incorrect to single out that set from the data for any statistical analysis: it is just another data selection error. We verified Haselhoff's analysis of the Hoeven formation; the results can be found in the article linked here: English, Italian. Briefly: Haselhoff's analysis is affected by some errors (mostly data selections); the corrected results show no reasonable evidence for BOLs.

"One can only speculate why Grassi decided no to include these data in his analysis."

It would have been the wrong place. The Hoeven report had never been published in a scientific journal, so there was no need to criticize it in a scientific journal. We felt that if we did, we might have implicitly credited the report with a scientific status it did not deserve. Moreover, our paper was already pretty long. So we published our analysis of the Hoeven case in the magazine of CICAP, Scienza & Paranormale (S&P N. 63 - Year XIII - Sep/Oct 2005), and put it online too.

"Finally, I would like to express my astonishment about the bombastic and denigrating style of Grassi's paper, in which he pretends to debunk a sensational claim. Anyone who reads my paper will agree that this was a mere comment to the work of the BLT team, suggesting some model adaptations and carrying ahead their hypotheses with a modified version, only to stimulate further study."

We have already commented Haselhoff's recent just-a-comment position, but an additional remark may be useful here. Haselhoff seems to think that we should not have criticized his paper because it is "a mere comment". Scientifically speaking, there is a precise difference between a peer-reviewed paper and a simple comment. A comment is the expression of the opinion (opinion, not science) of somebody about the work carried out by somebody else, with the bonds imposed by the editorial board of the review only (and no checking by the scientific board). Moreover, while a comment should enjoy the advantage of being not scientifically criticizable, as an expression of the free thinking of someone, at the same time it should not pretend to scientifically prove anything but a subjective personal thinking.

We already remarked that Haselhoff's "comment" is not a mere comment at all, since it contains new hypotheses, data, calculations and conclusions, just like a full paper, and so it is also susceptible of technical criticisms, as we did. But even if we accepted - and we do not - Haselhoff's opinion that his work should not be commented at all because it is - formally - not a full paper, he would run into another problem. We have just criticized Haselhoff's claim that his Hoeven report supplies the scientific evidence that was missing from his paper, but apart from our criticism, the Hoeven report has never been published in any scientific journal, not even as a comment. Coherently with his recent view that nothing less than a formal peer-reviewed full paper should ever be considered as an attempt to scientifically prove anything, he should have never mentioned his Hoeven report as a source of scientific evidence in the first place.

"In my opinion the style of Grassi's comment, as well as the propaganda related to it that he currently carries out over the internet and beyond is way out of proportion, and casts a dark shadow over his true intentions."

Propaganda? It is not us who perform data selections. We analyze available information and publish the results. We are interested only in claims on the paranormal and in facts that should back them. Therefore, we will not comment Haselhoff's style nor draw any conclusion about his true intentions (although readers may do): that is definitely beyond our interests.

Haselhoff's "comment" paper highlights a couple of very serious errors in the BLT paper it refers too, actually invalidating its conclusions. This is a serious criticism, however presented. Any researcher has the scientific right to criticize anybody else's work. Any serious researcher knows that rule; science would never work otherwise. The primary reason why a scientist must be so careful on his work is that he knows all too well that if he makes mistakes, somebody else will find them. Haselhoff was just in his right when he pointed out BLT's errors; so were we.

Moreover, we ourselves directly invited Haselhoff, and Levengood & Talbott on behalf of Haselhoff, to write an official reply and to submit it to JSE journal. We were not satisfied by the "rapid reaction" published by Haselhoff on the DCCCS website, because it was not a true scientific criticism but a simple opinion. We waited till now, expecting a scientific publication from both Haselhoff and Levengood & Talbott that deeply and with well founded reasons criticizes our article on the JSE.

We are not closed to any scientific criticism on our work, that obviously is at disposal of anyone who would like to investigate it. So far we have had no reply.

"Conclusion

Normally, authors of scientific communications are asked for comment before their work is criticized. This is not just a matter of courtesy, but it also avoids precious time being wasted. Any serious researcher would take the time to verify if there is some fact they have overlooked or possibly misunderstood before they rush to publications and press releases. I can only conclude that Mr Grassi has little or no experience in scientific communication. Open and honest correspondence, initiated by Mr Grassi and co-authors would have saved myself, them, as well as the editors of Physiologia Plantarum and the Journal of Scientific Exploration much unnecessary work. I do appreciate efforts that are dedicated to the pursuit of knowledge. However, I can't help suspecting from the course of matters that Mr Grassi was more dedicated towards disgracing me and my work rather than to performing honest and sound research."

Haselhoff's point would have some value if our paper had really been just a waste of time, i.e. if all points made in it had turned out to be groundless. On the contrary, all of our conclusions still hold. About the Nieuwerkerk data: not even the incident hypothesis would turn our work on this single point into a waste of time. At least, it would have lead to the identification of an error whose existence had never been acknowledged before, i.e. that the data set published in Haselhoff's book was completely wrong.

We did take the time to verify our analysis as deeply as possible. There is a reason why our paper has three authors: we discussed every single point to great lengths. We also asked several other experts for their opinions. This process took a lot of time, but it was well spent. The only waste of time we met during this study was the hopeless wait for most of the data we asked Haselhoff for. However, on two occasions (Grassi, April 2003; Russo, February 2004) we did try to discuss obscure points of his work with him, with no results. We just gave up.

Francesco Grassi
Ingegnere (Dr. Eng.), CICAP Experimentation Group

Claudio Cocheo
Centro di Ricerche Ambientali, Fondazione Salvatore Maugeri, CICAP Veneto

Paolo Russo
Programmer, CICAP Friuli Venezia Giulia

They all are members of the CICAP Study Team on Crop Circles


Go to:
SPECIALE Crops

 

 

Email:
crops@cicap.org

 

 

 

 

 

 

 

 

 

 

 

 

 


Inizio pagina

for info on this website and CICAP: info@cicap.org

© 2005 CICAP - All right reserved. Nessuna parte di questo sito puo' essere riprodotta senza il permesso scritto del CICAP.