The Troubled Future of Higher Education

The political scientists Gary King and Maya Sen have just posted an excellent working paper clearly outlining the major problems facing higher education: economic, political, sociological. The main thrust is that, although only 30% of the American population obtains a four-year college degree (thus leaving an untapped 70% who could finish college degrees), the higher education system is facing major constraints due to limited budgets and major technological advances. For example, online sites such as Khan Academy are effectively competing with universities, and for-profit universities are growing at a high rate. I'd add to their list the potential for big data analysis to displace the role of experts; I refer to the effect of sabermetrics on baseball journalists or data mining algorithms on marketers as possible canaries in the cage for academics. Regardless, King and Sen's paper is a must-needed beginning of a discussion about the future of higher education in the wake of profound social changes. After all, it was only a mere decade ago that Time and Newsweek were major cultural institutions in American life.

The Future of the Academy in 2032

Just before he died, for a few years I helped the great sociologist Dan Bell with using his computer, and as a result I got to know him very well. One thing I learned from him (besides the distinction between "criticism" and "critique") is the usefulness of prediction as an endeavor in itself (as opposed to explanation). In this spirit, I offer five predictions about the future of the academy in 2032:

1. First, despite opposition from many established institutions, there will be an enormous increase in open-source education. Classes on any topic will be available online for free, with lecture notes, videos, presentations, and chat services (with other students) available to anyone with a computer. Exemplars of this trend include MIT OpenCourseWare, Khan Academy, and videolectures.net. 
2. Second, academic publishing will be increasingly online, with peer review a continuous process. Rather than books and articles published at one time in paper form after a process of peer review, academic projects will be ongoing, process-oriented, available online, and subjected to a continual process of peer review. In essence, everything that academics produce will be works-in-progress, and updated when errors are noted. Early indications of this trend include the NBER archive and  arxiv.org.
3. Third,due to technological changes and increased monitoring of people's activity, academics will have to be adept with managing and analyzing big data. Common statistical methods will often be difficult to use on such large data sets, straining the computational capacities of computers. While not common in the academy yet, big data is one of the top buzzwords of 2012, and I expect this to spread to academic work relatively soon. An exemplar of this kind of academic work is the Google ngrams project. (One danger, however, is that private corporations might be hostile to information-sharing, and the values of profit-making may severely inhibit the availability of big data to academics.)
4. Fourth, big ideas will actually be in greater demand in the future. Precisely because there will increasingly be an excess of information, grand theories and master narratives will be increasingly desired to help guide attention, avoid fragmentation of different research traditions, and unify otherwise disparate theories. For example, Josh Tenenbaum's efforts at unifying artificial intelligence (which suffers from disciplinary fragmentation) with probabilistic graphical models is a promising endeavor.
5. Finally, the skills in demand will be increasingly modular rather than topical. For example, as part of the Cold War in the 1960s, the United States government funded various "area studies" programs to educate Americans on the traditions, customs, and practices of various geographic regions around the world. In the future, there will be less emphasis on this kind of topical knowledge, and greater emphasis on modular skills  such as critical analysis of any kind of texts or arguments, understanding the basic structures of any set of languages, and gathering and analyzing various kinds of qualitative and quantitative data. 

To the extent any of these predictions are correct, sociology is particularly well-suited to take advantage of these trends. Sociologists are generally supportive of the democratic, inclusive principles of open-source education and online publishing, and sociology has an unparalleled tradition of big ideas. Moreover, modularity is ingrained in the discipline; in fact, sociology is almost by definition a modular discipline, inasmuch sociology is an approach to a particular subject matter rather a particular subject matter per se.

McKinsey on Big Data

McKinsey has a full report (from March 2011) describing the meaning and potential impact of so-called big data. You can read the report here. One problem, which the authors of the report do not discuss in detail, is the that since so much of what constitutes big data will be collected by private firms there are possibilities of restricted information pockets. In other words, only certain private actors will have access to big data, and academics might very well be left very few big data sources.

Universal Limits in High-Dimensional Statistics

The MIT Center on Operations Research is hosting a talk tomorrow on universal limits in high-dimensional statistics. The basic idea is that, for all fields of empirical study from sociology to high-energy physics, some criterion for "statistical significance" is crucial for making decisions based on the data. (The current hunt for the Higgs Boson particle is in fact based on a modified criterion for statistical significance.) The problem, however, is that we are entering a world of big data, in which data structures have many dimensions, thus altering the potential usefulness of such criterion for statistical significance.

Big Data and the End of Theory?

An article in The Guardian gives appropriate caution to claims that data analysis (and only data analysis) is the solution for all or even most academic and research problems. As Max Weber observed in  his brilliant essay on objectivity in the social sciences, even the process of data analysis depends on values that cannot be empirically proven as right or wrong: "The 'objectivity' of the social sciences depends [..] on the fact that the empirical data are always related to those value-ideas which alone make them worth knowing and the significance of the empirical data is derived from these value-ideas. But these data can never become the foundation for the empirically impossible proof of the validity of the value-ideas."