So I’ve been learning Python by using iPython for analysing my twitter data. The iPython notebook provides a fantastic environment for doing this by letting you [...]]]> As I said in my last post, I was inspired by the talk at OSDC2011 by Dr Edward Schofield, Python for R&D to try out Python and in particular iPython. 

So I’ve been learning Python by using iPython for analysing my twitter data. The iPython notebook provides a fantastic environment for doing this by letting you write notes in between blocks of python code, and see the results from running the python on the same page.

I’m starting ipython by opening a terminal window in the directory I have my ipython notebooks and running:
ipython notebook --pylab inline which makes the matplotlib graphics appear inline (on the webpage) instead of in a separate window.

I tried a few different approaches to getting everything working on Mac OSX Lion. The Scipi-Superpack for OS X was the last I tried, and it seems to have got the last piece that I hadn’t got working via the other approaches, Pandas and scikits.statsmodels, working.

I’m using the dev version of iPython from GitHub. It is great that they have it setup so that each time I pull the updates they are available straight away without any extra install just by restarting the notebook.

I began by working out how to get data sets from mySql and from Apache Solr and then draw graphs of them using matplotlib. I used paired lists for this as that was what the matplotlib examples used. When I started trying to add time series of different lengths and with different gaps in the data I started to find the limitations of paired lists. Looking around for python time series libraries I found scikits timeseries which looked good, but then came across scikits.statsmodels and Pandas and decided to try them.

If you want to try running this code, I’ve linked the iPython notebook that these code snippets were taken from at the bottom of the post.

Convert pair of Python lists to Pandas series

Pandas makes it easy to convert the paired lists into a pandas.series object:

def convertListPairToTimeSeries(dList, cList):
    # my dateList had date objects, so convert back to datetime objects
    dListDT = [datetime.datetime.combine(x, datetime.time()) for x in dList]
    # found that NaN didn't work if the cList contained int data
    cListL = [float(x) for x in cList]
    # create the index from the datestimes list
    indx = pandas.Index(dListDT)
    # create the timeseries
    ts = pandas.Series(cListL, index=indx)
    # fill in missing days
    ts = ts.asfreq(pandas.datetools.DateOffset())
    return ts

Adjusting the Pandas series

I then made the two data sets (tweets received per day and tweets limited per day) have the same start and end and filled in all the missing days with 0′s or, where I knew that I had a data collection outage, with NaN.

# my data had lots of gaps that were actually 0 values, not missing data
# So I used this to fix the NaN outside the known outage
startOutage = datetime.datetime(2011,12,7)
endOutage = datetime.datetime(2011,12,8)
# set all NaN values to 0
tsFilled = tSeries.fillna(0)
# set the known outage values back to NAN
tsFilled.ix[startOutage:endOutage] = numpy.NAN

If the gaps in my data were all meant to missing values instead of 0, I could have just left the series as they were and used pandas.join instead of + to add them together

Truncating the data

I truncated the data, both to make the series the same length (although there are other ways to do this) and to remove partial days of data at the start and end of the periods.

# use slicing to change length of data
tSeriesSlice = tSeries.ix[startData:endData]
# use truncate instead of slicing to change length of data
tSeriesTruncate = tSeries.truncate(before=startData, after=endData)

Plotting the data

Basic graphing of a Pandas series is very straight forward:

tsFilled.plot();

It is easy to have multiple lines on the same graph, and to add titles and axis labels.

tsFilled.plot(label="original")
tsNew = tsFilled+(rand(len(tsFilled)))
tsNew.plot(label="adding")
tsNew1 = tsFilled.fillna(1)
tsNew1 = tsNew1 +(rand(len(tsNew1)))
tsNew1.plot(label="fillna+adding")
plt.legend(loc=3)
plt.title("Testing Panda Plotting")
plt.ylabel("Counts");

For more control over the layout, it is possible to pass the matplotlib axis in the Pandas plot statement.

# loc = MonthLocator()
loc = DayLocator(interval=2)
formatter = DateFormatter('%d %b')
fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(211)
plt.title("Testing Panda Plotting")
tsFilled.plot(label="original", ax=ax)
plt.ylabel("Counts")
plt.legend()
ax2 = fig.add_subplot(212)
tsFilled.fillna().plot(label="filled", ax=ax2, color='g')
plt.legend()
plt.ylabel("Counts")
plt.xlabel("2011")
ax2.xaxis.set_major_locator(loc)
ax2.xaxis.set_major_formatter(formatter)
labels = ax2.get_xticklabels()
setp(labels, rotation=80, fontsize=10);

For plotting multiple lines, it is probably better to add the series into a Pandas dataframe and then plot from that, but I’ll leave that for another day.

I’m new to python, matplotlib and Pandas, so I’d be very happy for any feedback about better ways to do things.

iPython Notebook for these examples

Download pandasTimeSeriesNotes.ipynb_.zip (58k)

Links

• iPython – http://ipython.org/
• iPython from GitHub – https://github.com/ipython/ipython
• Scipi-Superpack for OS X – https://github.com/fonnesbeck/ScipySuperpack
• Pandas – http://pandas.sourceforge.net/
• scikits.statsmodels – http://statsmodels.sourceforge.net/
• Pandas and StatsModels discussion list – https://groups.google.com/group/pystatsmodels
• matplotlib – http://matplotlib.sourceforge.net/
• scikits timeseries – http://pytseries.sourceforge.net/

When I looked out our dining room window this morning I saw hundreds of tiny praying mantises on the window.  The picture is from outside – at the top of the window.  You can see the egg case that the nymphs are coming out of. They were all over the window too.

We’re showing an installation called boidSong in the “Inspired” art exhibition at The Front Gallery in Lyneham from the 20th-25th August as [...]]]> I’ve been having a lot of fun recently collaborating with Eleanor Gates-Stuart and Barry Moon. We’ve setup a website for our project –  m o o n s t u a r t   – at moonstuart.net.

We’re showing an installation called boidSong in the “Inspired” art exhibition at The Front Gallery in Lyneham from the 20th-25th August as part of National Science Week.  

The opening is at 8pm on Wednesday 20th August. 

What is SciVee?

SciVee is about the [...]]]> The Public Library of Science (PLoS), National Science Foundation and the San Diego Supercomputer Center have released a new website where scientists can post videos. At this stage it allows the upload of videos related to articles published in PLoS journals. From their FAQ:

What is SciVee?

SciVee is about the free and widespread dissemination and comprehension of science. To learn more about SciVee link to our About Page.

SciVee is operated in partnership with the Public Library of Science (PLoS), the National Science Foundation (NSF) and the San Diego Supercomputer Center (SDSC). Please see our Partners page to learn more about them.

PLoS is a group of online peer reviewed journals using a new approach – the journals are free to access but charge a fee if you want to publish an article in them.  This fee covers the costs of publication and peer review – but the articles still need to pass peer review in order to be published.

I found this book very easy to read despite the complicated ideas it covers. The author has succeeded in making it an entertaining and accessible book.

It gives a good overview of quantum physics both from a historical perspective and the implications quantum physics has for our understanding of the world.

It doesn’t [...]]]>

I found this book very easy to read despite the complicated ideas it covers. The author has succeeded in making it an entertaining and accessible book.

It gives a good overview of quantum physics both from a historical perspective and the implications quantum physics has for our understanding of the world.

It doesn’t attempt to cover any of the mathematics or complexity. It explains any specialist terms at they are introduced, and uses good analogies to give you a feel for the topic.

If you are interested in physics then this is a great way to start looking at Quantum Physics.

I’m not sure why, but they always seem to be the best lectures we go to during the year. I think it may be because they go for longer than the 1 hour that most public lectures run for. The are [...]]]> If you are interested in Physics, then these Public Lectures are very
good.

I’m not sure why, but they always seem to be the best lectures we go to during the year. I think it may be because they go for longer than the 1 hour that most public lectures run for. The are scheduled to be from 8pm to 10pm but usually go for about 1.5 hours. So they can cover more detail on the selected topic.

http://kiosk.ph.unimelb.edu.au/news_events/july_lectures

The first one was last Friday night, and it was excellent. One every
Friday night at 8pm during July.

A list of all the events at Melbourne Uni is available here: http://events.unimelb.edu.au/ (tip: search for “Public Lectures”)

“Freakonomics is a book that will change the way you look at the world. It is not like an ordinary non-fiction book, it’s full of quirky facts and seemingly illogical truths. For example; which is more dangerous, a house with a pool or a house with a gun? In Freakonomics you’ll for everything from what schoolteachers and sumo wrestlers have in common to why drug dealers still live with their mums.”

The authors have a website http://www.freakonomics.com/ which has some updated information, reviews and a link to a student guide. The student guide provides an interesting way to have a second look at the book. There is also a teachers guide available if you apply for a userid/password from the Harper Academic website.

Planetary scientists have developed a theory that can explain three outstanding puzzles about our solar system: Why do the giant planets have eccentric and tilted orbits? How did Jupiter get its Trojan asteroids? And what caused the Late Heavy Bombardment some 700 million years after the Earth and Moon were [...]]]> An interesting article on physicsweb:

Planetary scientists have developed a theory that can explain three outstanding puzzles about our solar system: Why do the giant planets have eccentric and tilted orbits? How did Jupiter get its Trojan asteroids? And what caused the Late Heavy Bombardment some 700 million years after the Earth and Moon were formed? The answer seems to be that all three phenomena seem to be a direct result of Saturn and Jupiter shifting their orbits in the early solar system.

Read the full article.