Kevin and Oliver's GC Index

Brad Anderson wrote a java application to run on his server that pulls GCP/EGG data in near real time to display a summary statistic in a color index (see below). It has been revamped by Kevin Laighleas and Oliver McDermott. The application runs on A dedicated server, and is linked from the shaded dot on the GCP homepage. Here are some explanatory extracts from Brad's original description of the process.

The server calculates the probability of Chisquare (the squared Stouffer Z, or Z²) on a second time frame basis (in real time) following the GCP analysis procedures. Then a Mac Widget (mini application) pulls the statistical analysis from the server and presents the user with a colored button representing the probability.

The system is designed to avoid having each individual instance is pulling data from the GCP servers. The processing would take too long, and if this widget got popular, it could overload the servers.

The program runs the statistics continuously minute by minute. It then dumps a file of the second by second probability for the minute into a web directory. Each client downloads this probability file for display. This way, the dedicated machine has the tough job of statistical analysis, while the GCP servers are isolated from the potential high volume that this could generate, and the clients don't have to spend any time processing.

The server runs 10 minutes behind. It stores the prior minute of statistics while it pulls the current minute. The clients are buffered with another minute to allow the data to be downloaded asynchronously while the client displays older data. This way there is no stopping or blocking when the data is grabbed. Each has its own one minute buffer, putting the client 12 minutes behind the data generation.

The program creates a Chisquare for the most recent hour (the sum of 3600 seconds' Z² values). The probability of the Chisquare is then calculated. This value, representing the average departure of an hour of data from expectation, is recalculated each second in a sliding window, adding a new second and dropping the trailing second. Thus, the display changes very slowly, and gives an impression of the state of the network averaged over a substantial period.

The color coding represents the level of coherence or correlation among the eggs, which is reflected in the probability of the Chisquare. The expected level is about 50%, and big shifts in either direction are notable. The GCP's formal testing looks for increased interegg correlation, which is represented here by the warm colors, orange and red.

• Blue starts to fade in at 90% and above.

• Green represents about 50%

• Yellow starts fading in from green at 40%.

• Orange fades in at 15% or so.

• Red is 5% which is regarded as "significant".

• Bright red is 1%, or odds of 1 in 100.

Please note that this display is not intended or expected to be interpreted as if it were a formal analysis using large amounts of data. In particular, it should not be construed as a preliminary indicator of what might be happening with the "global consciousness", and it certainly cannot be used as an early warning system about possible events.

It is just a composite display of a statistic that reflects the relative intercorrelation of the eggs, and it cannot be reliably interpreted. The reason is that the best data we have, based on sophisticated analysis, shows that the signal to noise ratio is too small for any single event or brief data segment to be meaningful. What we see in analysis of a few minutes or hours is partly signal but mostly noise, and the noise cannot be distinguished from a possible signal -- until we compound 30 or 40 events or the equivalent of many hours to days of data. The dot is an interesting display of the current state, and is in fact the product of real-time analysis, but it isn't suitable for drawing interpretive conclusions.

All that said, we can make some suggestions about the meaning of the colors:

• Blue: Extremely small network variance. Suggestive of deeply shared, internally motivated group focus. The index is above 95%

• Aqua: Small network variance. Probably chance fluctuation. The index is between 90% and 95%

• Green: Normally random network variance. This is average or expected behavior. The index is between 40% and 90%

• Yellow: Slightly increased network variance. Probably chance fluctuation. The index is between 10% and 40%

• Orange: Strongly increased network variance. May be chance fluctuation, with the index between 5% and 10%

• Red: Extremely large network variance. Suggests broadly shared coherence of thought and emotion. The index is less than 5%