The manner in which scientific discoveries are made is changing dramatically. In the not too distant past, a scientist or small group would make some observations. Thinking about them, and comparing them to other observations would lead to a theory which would be tested for reality. For example, before about 1900 scientists thought the Universe was filled with an ether that allowed light to travel through space, think like water but not visible. Light, the planets, and everything else were thought to travel through this ether. It was hypothesized that we could measure this either by measuring how the speed of light changed when the Earth went in one direction through the ether in say January and the opposite direction in June. The measurement failed, there was no difference. Einstein thought to himself, what if the measurement is correct. The result was the theory of relativity. We had a small data set and some thought which led to a major scientific breakthrough.

We now have extremely large easily accessible data sets and artificial intelligence to sift through it in very little time. Just five years ago, analysts estimated that the world generated approximately 4.4 trillion gigabytes of data. That's 4.4 trillion computer hard drives. As this information has become available, data scientists have developed tools to extract insights from it. These tools are rapidly replacing the people who once served as the gatekeepers to new discoveries.

Big data will soon change the face of astronomy. In an effort to accelerate astronomical discoveries, engineers have built the mother of all telescopes, called the Large Synoptic Survey Telescope, or LSST. The LSST will be equipped with a massive, three-ton digital camera that will be able to cover the entire southern sky in just three days. It will generate petabytes of data that will be immediately distributed around the world for instant analysis.

In the past, astronomers were lucky to reserve a few nights worth of observations on one of the few high-powered telescopes in the world. This process, if well planned and efficiently executed, could allow the astronomer to observe a few dozen or a few hundred objects. In contrast, the LSST is expected to identify 20 billion galaxies and 17 billion stars in its first decade of operation. Artificial intelligence will play a major role in identifying these objects, thereby changing the relationship between the scientists and their data.

How will the new data frontier ultimately affect scientists? Some claim that science may soon be dominated not by people but instead by computers. Even the revered practice of observation and prediction, as described above,—core elements in the scientific process and once the domain of the lone genius—is being challenged. With the right software, ever-cheaper hardware, and the free availability of data on the web, even a non-expert in the field can identify relationships within large data sets.

We're beginning to see the impact of these efforts in every area of research. In my opinion, it will still require the intelligent, well educated, scientist to determine the veracity of the results. But, he/she will be inundated with discoveries, not data.

Although we understand how big data is already changing how research is being done, there are many questions to be answered about how we can respond to these changes. There is a need for in-depth conversations about the ethics of data collection, analysis, and usage, how we will communicate discoveries when the process is largely computer-driven, and how the current scientific infrastructure system will need to adapt to accommodate these changes. We are now entering a truly dynamic exciting time.

Much of this article was copied, with permission, from a letter from Dr. Jamie Vernon, the CEO of Sigma Xi, the international research honor society.