Data Science: Republican & Democratic Conventions

In the past few weeks, the two major political parties in the United States of America held their national conventions. While I couldn't listen to all the speeches, I followed the news and paid attention to the overall scene. After they were done, I decided to grab the speeches of the major speakers and see if I could find any obvious trends in their word choices, similar to what I did with my Twitter project. In this blog post, I'll discuss what I can see in the data. You can find my data and all my scripts at this GitHub repo.

Data Science: The Divided States of America

In the prior two posts, I have described how I gathered twitter data from @HillaryClinton and @realDonaldTrump, how I ran a sentiment analysis on the individual tweets, and how I performed a principal component analysis on the most commonly used words. Today, I’ll tie everything together and describe how I created a model to predict whether a given tweet belongs to either of the two candidates.

Data Science: Principal Component Analysis of Twitter Data

As described on my last blog post on this topic, I've been tracking tweets from the US presidential candidates, Hillary Clinton and Donald Trump. I've looked at the top words they used and the sentiments expressed in their tweets given their word choice. However, some words are used with others almost all the time, a notable example being a slogan like Make America Great Again. As such, it may be beneficial to look at groups of words rather than individual words. For that, I took an approach applying a Principal Component Analysis. Below I describe what this is, how I used it, and what it reveals. Do note, however, that I'm applying things I learned in astronomy to this problem rather than taking courses specific to text mining. It may be that there are better tools out there than what I've used.

Data Science: Presidential Candidates on Twitter

Over the past few months, I've been working on a little hobby data science project to explore twitter data with regards to the upcoming presidential election in the United States. The project has evolved quite a bit and detailing it in full is beyond the scope of a single blog post. As such, I've decided to split it into (at least) 3 posts. This post is the first of the series and will go over the basics of gathering data from Twitter and doing some simple text mining. The second and third posts will discuss more details of the project and show some neat visualizations I've created. I'll release all my code after the third post for any curious coders. For now, let's get started seeing what Hillary Clinton and Donald Trump's Twitter accounts are talking about.

Chile on September 18

September is a month of history in Chile. Back on September 11, we saw the remembrance of the military coup that overthrew Allende and established the Pinochet dictatorship. Today, however, we go further back in history: to September 18, 1810 and the first steps in Chile's independence. From my readings, it looks like there were many notable dates throughout the process for independence, but September 18 is celebrated as the Day.

Chile on September 11

In the United States of America, September 11, 2001 (aka 9/11) is remembered as the day terrorists hijacked several aircraft and crashed them, most famously into the Twin Towers of the World Trade Center in New York City. While those events were tragic, it's sometimes easy to forget, particularly in this political climate, that the US is not the only country of the Earth. There is a whole world out there for whom the date may or may not have special significance.
Chile is one country that remembers something similar for September 11.

Science Education Standards in America

I recently read a very interesting, and alarming, Scientific American article on statewide science standards in the US. This picture says it all:

Credit: The State of State Science Standards 2012

The figure comes from a report from the Thomas B. Fordham Institute on the standards for K-12 education used in the various states. The overall conclusion is that the science standards for the majority of the States are mediocre to awful.
These standards are what's used to build curricula through the various states and grade levels. For example, I looked up the science standards for California and browsed through to the astronomy related ones. For sixth grade, here's what it has to say:

The solar system consists of planets and other bodies that orbit the Sun in predict­able paths. As a basis for understanding this concept:
  a. Students know the Sun, an average star, is the central and largest body in the solar
system and is composed primarily of hydrogen and helium.
  b. Students know the solar system includes the planet Earth, the Moon, the Sun,
eight other planets and their satellites, and smaller objects, such as asteroids and
comets.
  c. Students know the path of a planet around the Sun is due to the gravitational
attraction between the Sun and the planet.

So, a sixth grade teacher would be using these guides to make lessons that teach these concepts. I think those are pretty key concepts that everyone, even non-astronomers, should know. What the study presented is that some states have very clear and well-thought out guidelines. California is one such state and is graded 'A'. The report for Colorado follows and it's ranked at 'D'. Here's what I found for the science standards for Colorado (I'm looking at the eight grade Earth Science standards):

1. Weather is a result of complex interactions of Earth's atmosphere, land
and water, that are driven by energy from the sun, and can be
predicted and described through complex models
2. Earth has a variety of climates defined by average temperature,
precipitation, humidity, air pressure, and wind that have changed over
time in a particular location
3. The solar system is comprised of various objects that orbit the Sun
and are classified based on their characteristics
4. The relative positions and motions of Earth, Moon, and Sun can be
used to explain observable effects such as seasons, eclipses, and Moon
phases

Compare, in particular, point 3 from Colorado to point b from California. Both are talking about the same thing, but one is much more vague. This vagueness in what exactly constitutes a valid lesson in the various states is one of the points the report raises. Unclear guidelines are basically meaningless and useless when defining a curriculum or trying to assess a student's understanding. You may argue that they give the instructors greater flexibility in what they teach, but really- there are some basic facts that students should learn and this should be made explicitly clear.

Another thing mentioned in the article and the report is the undermining of evolution. You've probably heard about this on various news sources. This is part of a growing trend in the US to make laws that prevent the teaching of evolution or enforce teaching intelligent design, creationism, or other alternatives. One troubling example is recent changes in New Hampshire: teachers are required to provide alternatives to any lesson if a parent dislikes it. This means that a high school student could in principle graduate having avoided learning about evolution, the Holocaust, contraception, or even gravity, all because his or her parents thought those were touchy subjects.

While there is a religious undercurrent to this, I think it's more of a misunderstanding of science and a fear of the change it can drive. I went to a private, religious high school in Puerto Rico and I learned about evolution my science classes, as it should be. I don't doubt that there are many schools out there that have no problem with teaching evolution, regardless of whether or not they are religious schools. The problem, I think, is that most people, and the politicians in charge, have no idea what evolution is and think that scientists are somehow out to get them and shatter their beliefs. Not teaching, or undermining, evolution just reinforces this idea and makes things worse.

All this talk against science makes me think Americans want to believe, not to know or understand. Science and mathematics give us tools we can use to understand the world around us and figure out things on our own. While you may not think this negligence regarding science education is a big issue, in the long run, it will be. If this continues on the same trend, less and less emphasis will be given to providing a meaningful science and mathematical education to the following generations. Student's will still graduate, but what they will require for that degree will be ever less. All of the advances of technology we enjoy, like cars, iPods, the Internet, satellite television, all come about thanks to our understanding of science. When I see such attacks against eduction (or these poor science standards), I have to wonder: why don't the people in power want us to continue to advance in technology and in our understanding of the universe? Are they afraid of what we might learn, or what we might do if we can think for ourselves?

SOPA: Stop Online Piracy Act

As you have undoubtedly heard by now, many websites on the internet, including Wikipedia, were blacked out today in protest of the Stop Online Piracy Act (SOPA) and the Protect IP Act (PIPA). The goal of the bill is to go after pirates distributing copyrighted material, however, the provisions in the bill severely encroach on freedom of speech liberties. Basically, it allows for the censorship of websites and material on the internet.

I'll direct you to a few places to get more information:
Here's a good summary of what's going on with ways on how you can sign up against SOPA: http://americancensorship.org/
Here's a great video explanation.
Here's some more information as well.
And here's the Wikipedia article on SOPA and Google's End Piracy, Not Liberty page.

I am opposed to the bill, but as a citizen abroad I'm not sure how well I am represented. Still, it's good to hear that my comrades in the States are also against this and have been calling their representatives to make their voices heard.

If media corporations want to fight piracy, here's what I suggest: provide affordable alternatives worldwide. You remember back in the day when people would download music illegally? Well, if you want some music now, you can use Amazon, iTunes, and a number of other services to get your music legally and reasonably priced. With easy alternatives like that, there's no incentive for average people to resort to piracy.
Ideally, this should also extend worldwide. Our society has become globally connected, we hear about news and events from all over the place and can talk to people across the planet in an instant. Yet it is annoying when I go to Hulu and find out I can't watch some TV episodes because I am in Chile. The reasoning behind this is copyright restrictions on different countries. Perhaps when you start having piracy problems, these corporations should look to the source. I would say the solution is not to tighten your grip and lash out at your consumers, but to explore new ways to release your products.

This is part of some drastic and alarming changes taking place in the US ever since 9/11. It's as if people collectively (or, perhaps more accurately, those in power) are afraid and are happy to give up freedom and basic rights in exchange for a little safety. Remember the days when you didn't have to take of your shoes at the airport to board a flight? It seems to me that rather than 'innocent until proven guilty', we've been changing it to 'assume they're guilty... just in case'.
While this isn't 100% related to SOPA/PIPA, I think all of these things are pointing to a very dark future ahead of the US in terms of civil liberties. I recommend reading 1984 by George Orwell, to see how a heavily censored police state works.