Changing Tides - Brenda Moon

Another step in my python journey – virtualenvwrapper

Brenda — Wed, 14 Mar 2012 12:24:32 +0000

I’ve been playing with the MakeHackVoid Sun computer (Morphia) to see how much faster some of my data mining programs run when given access to 64G of memory instead of 4G (not to mention the processors). The short answer is “a lot”!

So I started thinking about off loading the last part of the backend, which is the ipython notebook server. I don’t think that ipython notebook is putting much load on my computer, especially compared to what mysql and solr were causing. But I’m just interested to see if it is possible.

It would mean only opening a single ssh tunnel for the notebook instead of two for mysql and solr

For ipython notebook I need to install a much more recent ipython and then all the python libraries that I’m using. To keep the server stable, I’ll need to install all of these in my user space. Angus suggested that I look at virtualenvwrapper. Last night I installed it on Morphia and read these useful introduction pages:

Looks like it will do what I want. I’ll report again when I’ve tried to use it – hopefully soon!

Learning Python – iPython, matplotlib and Pandas

Brenda — Sun, 01 Jan 2012 08:40:54 +0000

As I said in my last post, I was inspired by the talk at OSDC2011 by Dr Edward Schofield, Python for R&D to try out Python and in particular iPython.

So I’ve been learning Python by using iPython for analysing my twitter data. The iPython notebook provides a fantastic environment for doing this by letting you write notes in between blocks of python code, and see the results from running the python on the same page.

I’m starting ipython by opening a terminal window in the directory I have my ipython notebooks and running:
ipython notebook --pylab inline which makes the matplotlib graphics appear inline (on the webpage) instead of in a separate window.

I tried a few different approaches to getting everything working on Mac OSX Lion. The Scipi-Superpack for OS X was the last I tried, and it seems to have got the last piece that I hadn’t got working via the other approaches, Pandas and scikits.statsmodels, working.

I’m using the dev version of iPython from GitHub. It is great that they have it setup so that each time I pull the updates they are available straight away without any extra install just by restarting the notebook.

I began by working out how to get data sets from mySql and from Apache Solr and then draw graphs of them using matplotlib. I used paired lists for this as that was what the matplotlib examples used. When I started trying to add time series of different lengths and with different gaps in the data I started to find the limitations of paired lists. Looking around for python time series libraries I found scikits timeseries which looked good, but then came across scikits.statsmodels and Pandas and decided to try them.

If you want to try running this code, I’ve linked the iPython notebook that these code snippets were taken from at the bottom of the post.

Convert pair of Python lists to Pandas series

Pandas makes it easy to convert the paired lists into a pandas.series object:

def convertListPairToTimeSeries(dList, cList):
    # my dateList had date objects, so convert back to datetime objects
    dListDT = [datetime.datetime.combine(x, datetime.time()) for x in dList]
    # found that NaN didn't work if the cList contained int data
    cListL = [float(x) for x in cList]
    # create the index from the datestimes list
    indx = pandas.Index(dListDT)
    # create the timeseries
    ts = pandas.Series(cListL, index=indx)
    # fill in missing days
    ts = ts.asfreq(pandas.datetools.DateOffset())
    return ts

Adjusting the Pandas series

I then made the two data sets (tweets received per day and tweets limited per day) have the same start and end and filled in all the missing days with 0′s or, where I knew that I had a data collection outage, with NaN.

# my data had lots of gaps that were actually 0 values, not missing data
# So I used this to fix the NaN outside the known outage
startOutage = datetime.datetime(2011,12,7)
endOutage = datetime.datetime(2011,12,8)
# set all NaN values to 0
tsFilled = tSeries.fillna(0)
# set the known outage values back to NAN
tsFilled.ix[startOutage:endOutage] = numpy.NAN

If the gaps in my data were all meant to missing values instead of 0, I could have just left the series as they were and used pandas.join instead of + to add them together

Truncating the data

I truncated the data, both to make the series the same length (although there are other ways to do this) and to remove partial days of data at the start and end of the periods.

# use slicing to change length of data
tSeriesSlice = tSeries.ix[startData:endData]
# use truncate instead of slicing to change length of data
tSeriesTruncate = tSeries.truncate(before=startData, after=endData)

Plotting the data

Basic graphing of a Pandas series is very straight forward:

tsFilled.plot();

It is easy to have multiple lines on the same graph, and to add titles and axis labels.

tsFilled.plot(label="original")
tsNew = tsFilled+(rand(len(tsFilled)))
tsNew.plot(label="adding")
tsNew1 = tsFilled.fillna(1)
tsNew1 = tsNew1 +(rand(len(tsNew1)))
tsNew1.plot(label="fillna+adding")
plt.legend(loc=3)
plt.title("Testing Panda Plotting")
plt.ylabel("Counts");

For more control over the layout, it is possible to pass the matplotlib axis in the Pandas plot statement.

# loc = MonthLocator()
loc = DayLocator(interval=2)
formatter = DateFormatter('%d %b')
fig = plt.figure(figsize=(8,4))
ax = fig.add_subplot(211)
plt.title("Testing Panda Plotting")
tsFilled.plot(label="original", ax=ax)
plt.ylabel("Counts")
plt.legend()
ax2 = fig.add_subplot(212)
tsFilled.fillna().plot(label="filled", ax=ax2, color='g')
plt.legend()
plt.ylabel("Counts")
plt.xlabel("2011")
ax2.xaxis.set_major_locator(loc)
ax2.xaxis.set_major_formatter(formatter)
labels = ax2.get_xticklabels()
setp(labels, rotation=80, fontsize=10);

For plotting multiple lines, it is probably better to add the series into a Pandas dataframe and then plot from that, but I’ll leave that for another day.

I’m new to python, matplotlib and Pandas, so I’d be very happy for any feedback about better ways to do things.

iPython Notebook for these examples

Download pandasTimeSeriesNotes.ipynb_.zip (58k)

Great talk at OSDC2011 – Python for R & D – Edward Schofield

Brenda — Sun, 01 Jan 2012 08:11:34 +0000

I went to OSDC2011 at ANU in November. I had a great time, met a lot of interesting people and heard some great talks.

This talk by Dr Edward Schofield – Python for R&D has inspired me to try using Python in my data analysis.

Twitter changed to SSL only for streaming API today

Brenda — Fri, 30 Sep 2011 03:58:01 +0000

This morning my Twitter data collection program suddenly started failing to connect. I’m using the the excellent twitter4j library for connecting to Twitter.

The error was “Connection Refused” with this response:
TwitterException{exceptionCode=[b5e7486f-24943238 b5e7486f-2494320e], statusCode=-1, retryAfter=-1, rateLimitStatus=null, featureSpecificRateLimitStatus=null, version=2.2.4}

I found out that Twitter has turned on only accepting SSL connections for connecting to streams today. (https://dev.twitter.com/blog/streaming-api-turning-ssl-only-september-29th)

I tried setting builder.setUseSSL(true) in Twitter4j, but that didn’t fix the problem. There is a new snapshot build of twitter4j that does fix it (2.2.5-SNAPSHOT). It is available for download from http://twitter4j.org.

I’m using Eclipse Helios and Maven and had some trouble working out how to get the SNAPSHOT. In the configuration I have, it picked up the snapshot of twitter4j-stream-2.2.5-SNAPSHOT.jar, but not the twitter4j-core-2.2.5-SNAPSHOT.jar. I tried a few different things to make it get the core snapshot which didn’t work, but then found that disabling the releases in the repository definition worked:

   
      twitter4j.org
      twitter4j.org Repository
      http://twitter4j.org/maven2
      
         false
      
      
         true

Not sure why the twitter4j-stream snapshot was downloaded but not the twitter4j-core without changing the POM. But with this change, my data collection is working again although I’ve missed a few hours of data.

Asteroid Discovery since 1980 – video by British astronomer Scott Manley

Brenda — Sun, 29 Aug 2010 06:31:15 +0000

An amazing video of Asteroid discovery since 1980. It is best viewed in high resolution at YouTube. There is an interesting article on the Daily Mail UK about it too.

Videos from MIT Hyperstudio’s 2010 humanities + digital Visual Interpretations conference

Brenda — Fri, 27 Aug 2010 00:54:41 +0000

The conference overview video which includes a brief moment of me demonstrating “Hearing the Unseen”:

Videos of the keynote speakers are available here MIT World – HyperStudio and the other talks are available here MIT TechTV – humanities + digital Visual Interpretations conference May 2010.

EarTrading available free in iTunes store

Brenda — Wed, 18 Aug 2010 04:55:10 +0000

EarTrading is now available in the iTunes store. It is a free application for iPhone. The website for Ear Trading is http://eartrading.moon.net.au. Please try it and tell me what you think.

Here are some photos of Barry and me demonstrating the earlier version of EarTrading called “Hearing the Unseen” at the MIT Humanities + Digital Visual Interpretations Conference 2010.

Brenda demonstrating Hearing The Unseen

Barry demonstrating Hearing the Unseen

Ear Trading

Brenda — Thu, 22 Jul 2010 06:37:56 +0000

Hearing the Unseen has evolved into an iphone game and generative music machine called Ear Trading. It is now ready for beta testing after which it will be released as on the iTunes store as a free application.

The website for Ear Trading is http://eartrading.moon.net.au

If you are interested in beta testing the application, please send your iPhone UDID (Unique Device Identifier) to eartrading@moon.net.au.

Note: To find your UDID, open up your phone in iTunes, click on your serial number (it should change to your UDID), then copy the UDID (using Edit > Copy or pressing Cmd/Ctrl-C)

Hearing the Unseen

Brenda — Mon, 19 Jul 2010 02:23:43 +0000

This is based on a Poster/Demo presented at the “Humanities + Digital Visual Interpretations Conference” hosted by HyperStudio – Digital Humanities at MIT 20-22 May 2010.

Barry Moon
Arizona State University
Brenda Moon
The Australian National University

This project started as a exploration of data sonification techniques. The abstract published in the conference program is a testament to this. As it progressed, the idea of using real-time data to create a game for mobile devices became more alluring. More specifically, a game where sound plays a major role in decision making, or even a game that produces interesting music with minimal interaction. For this application, the meanings of the data become far less important than its trends and time basis. We are using stock market data which wakes up and goes to sleep at fixed times of the day. Music, being predominantly time based, suits this kind of predictable framework upon which to drape its material. Although the creation of a game is the direction our sonification research has taken us, similar techniques could be applied to data to not only reveal details, but make the exploration of those details interesting and fun.

As you will see in the conference program, our early model was programmed in Processing and Max/MSP. While this gave us a great deal of flexibility over visualization and sonification, it would not be usable on mobile devices in the near future. For our game, the iPhone/iPad was chosen as the device, and Unity as the programming solution. Choosing to use the iPhone/iPad over Macintosh should generate a greater amount of feedback from users to aid in development and create a stronger sense of intimacy with the interface via touchscreen.

Most approaches to producing sound in interactive media follow precedents set in film. So far, in our work, we have combined two sonic elements traditionally used to enhance visual media: music and “off-screen” sound. Music reveals details in the data in two ways:

1, Amount of rhythmic activity relates to a share’s trading volume – the greater trading volume per sample (6 seconds), the greater the number of beats per measure

2, Pitch change across the duration of the “measure” relates to price changes of the share -downward pitch sequence for falling share price and upward pitch sequence for increasing share price.

Figure 1: Typical bar of music with moderate trading volume (9 beats) and share increase in share price.

When data is off-screen, it continues to be heard, spatialized according to its position in the corresponding visual space:

Figure 2 – Position of sounds relative to visual space

Figure 2a: Game View

Figure 2b: Top View

Our main concern in creating a musical sonification (as it is in most music creation) is the balancing of repetition with variation. We generate music statistically by choosing beats via weighted probabilities. This extends a drum machine programming analogy, which if simplified (to create really horrible techno for example), the kick drum is heard on the downbeats and snare drum on upbeats.

Figure 3: Beat Tables

Audio samples are triggered at the statistically chosen beats. Audio synthesis would allow greater sonic variety, but there are no methods presently available in Unity. Sixteen percussive samples of different loudness were created for each sound. There needed to be some balance between percussiveness (which aids in direction perception), and “pitchedness” (since we are using pitch as a perceptual referent for price change). Samples needed to be quite short (less than one second), again because of limitations in the Unity environment.

We ultimately wanted to call our game “Our Tax Dollars at Work”, but found some of the bailout recipients have extremely low trading volume. Instead we chose shares with large trading volume (which does include CitiGroup and Fannie Mae), and our working title is now “b-trade”.

Enhancements we would like make to our game before letting it loose on the public (as much as iPhone/iPad users represent “the public”) are enhanced visual interface, greater variety of sounds, the potential for users to design their own “beat tables”, and for the game to morph between multiple “beat tables” to create greater musical variety. We would also like the user to be able to choose which company’s shares they follow.

Max/MSP: http://www.cycling74.com
Processing: http://processing.org/
Unity: http://unity3d.com/

An Atlas of Now

Brenda — Fri, 29 Jan 2010 07:14:42 +0000

Will Grant and I have written an article on the Diffusion blog titled “An Atlas of Now” describing the project I’m working on at CPAS.

We are in the process building a system – using both established and novel open source intelligence methods – to gauge, assess and display changes in the online discussion of science and technology issues, controversies and dangerous alternate conceptions. Our goal is to develop a science communication ‘atlas of now’: revealing geographical and temporal changes in attitudes to science and technology issues throughout society.
(An Atlas of Now)

We will be presenting a paper on this project at the Australian Science Communicators National Conference February 2010.