Why would you do that?
Since a couple of years I have been using stylometric methods to analyse texts. I learned to use the great stylometric tool Stylo (written in R) at the European Summer School of Digital Humanities in Leipzig from two of the developers: Maciej Eder and Jan Rybicki.
Some months after I started my PhD as a member of the junior research group “Computational Literary Genre Stylistics” CLiGS, guided by Christof Schöch, at theat the Computerphilologie Professorship (hold by Prof. Jannidis) at the University of Würzburg, Germany. I was told that I had to learn Python because that was the programming mother tongue of the department. And I did so. Since then, many of my projects are a mix of very basic R script that call Stylo, and other more sofisticated scripts in Python that make the preprocess and the evaluation.
I am not the only person in this R-Python situation; actually in the last years at least two tools for Stylometry have been written in Python: Pystyl and PyDelta. So, why do I keep working with Stylo if I know more Python? For several reasons:
- Stylo is very well documented (installation, preparation of the corpus, general use…)
- It has a mailing group where you get answers and help
- It has been tested by hundreds of researchers
- The developers teach about the tool
- And they use the feedback of these workshops to improve Stylo (I have seen Maciej speed-coding some changes in Stylo during the class, uploading to CRAN, and asking the people to update Stylo)
- Because my PhD-tutors recommend me to do so
My stylometric tests are becoming more and more complex so it is starting to be a pain to jump all the time between two groups of scripts. I knew that one can use other programming languages inside Python, so I thought it was worth a try to see if it was possible to use R and Stylo in Python.
This blog post and its sibling Notebook (that you can download as a Git Repository with the corpus and the output data) are the first findings. I would be really happy to receive opinion and feedback.
The module that we are going to use is rpy2 https://rpy2.readthedocs.io/en/version_2.8.x/, which allows you to work with R in Python. Since it is very possible that this module it is not in your computer, you have install it, for example using pip3 (more info in its documentation: https://rpy2.readthedocs.io/en/version_2.8.x/overview.html#installation):
- sudo pip3 install rpy2
That was not difficult, but to make it work was. After some time I realised that the problem was the version of R in my computer. Although the documentation of rpy2 says that a 3.0 version of R should be ok, it was not. Updating R in Ubuntu was trickier than expected, so I uninstalled and reinstalled R and Stylo again, making sure that R’s version was higher than 3.0. I am currently working with 3.3.
So, enough talking, if you have already installed rpy2, let’s import it:
I am not going to explain how exactly rpy2 works (because it is not the poing of this notebook and because I couldn’t). Let’s just say that whenever we see anything starting with R., it will be a R object that we can call from Python. Example:
Stylo in Python
In the same way we can call Stylo in Python:
Maybe it gives us a warning messages RRuntimeWarning: I think the problem is in the kind of answer that Stylo gives you in command line of R while running, that cannot give you in the same ways in Python. Does anyone know how to fix that?
In the repository of this Notebook you can find a subfolder with one Spanish corpus of the CLiGS Textbox (https://github.com/cligs/textbox), prepared for stylometric tests. So I will define the path just as the current folder and I will call Stylo without the graphical user interface (if I would need the GUI we would just work in R!).
It is cool to see the answers of Stylo in a Jupyter Notebook running on Python, right?
When it is finished, a pop-up window from R will appear with the classic dendrogram that we all know:
Now, how can I define the arguments for Stylo? Because, as explained in the documentation of Stylo, the arguments for maximum and minimum MFW are called mfw.min and mfw.max. Let’s try that:
Python complains: it doesn’t expect a dot in a variable name. The grammar of R and Python are not compatible. For this cases the documentation of rpy2 (http://rpy.sourceforge.net/rpy2/doc-2.2/html/robjects_functions.html) recommends to pass the arguments as a python dictionary in which the keys are strings with the names of the arguments in Stylo. Example with a couple of arguments:
Or we can define pass arguments for the kind of analysis, output that we want, the size of the n-gram…:
Now we have in our folder all the files that we have asked: png, distance table, features used… Nice!
But what if I want to work further with this data in Python?
Using the data from Stylo in Python
In the cell above I have called stylo() and saved its output in a variable called I_love_this_stuff (following the documentation of stylo 😉 ):
As we see, this variable is a ListVector of length 9. Each of these items contain different information from the analysis I have done. Let’s print the 100 first characters of the first items:
The first item contains actually the distance matrix:
As we see, this object is a matrix in R. Working in Python we would be happier with a Pandas Dataframe. For doing that, we convert first the matrix to a Numpy array, we use this array to load I_love_this_stuff to the dataframe, and we pass the names of the rows and the columns.
There you have your beautiful Delta Matrix of your corpus as Pandas Dataframe, using Stylo but working only with Python scripts. Yey!
This is just a try. Many things could be done in different ways, I have probably overseen things, maybe there are better ways to deal with this Python-R problem… So, please, let me know your thoughts (email, twitter, comments in the blog post…). Thanks in advance and thanks to Christof for his feedback about this Notebook!