Harvesting Codeplex Statistics

Update: The method described below does not work anymore. The page on CodePlex that was POSTed to was closed to external queries and so doesn't allow CORS any longer... Alas, it was fun while it lasted!

For a while now, all MOBZystems code is hosted on Codeplex. That has really proven to be a good move, if only for the download  statistics Codeplex provides for each project. There is a Statistics page containing detailed historic information about the project, including graphs. That's fascinating information.

Harvesting Codeplex statistics for a single project

But of course, I need (want?) more: an aggregated view of all our projects on Codeplex. But there's no way to get that from Codeplex - that is: not from the web site itself. The number of page views, visits, and downloads is shown on every project page, though, for the last 7 or 30 days, or the lifetime of the project. That information is harvested using the following Javascript code:

$(document).ready(function () {
  $('#PageActivity .Link').click(function () {
    var val = $(this).attr('d:value');
    
    $.post(
      'http://runnet.codeplex.com/stats/getActivity',
      { period: val },
      function (data) {
        $('#PageActivity span').hide();
        $('#PageActivity .Link').show();
        $('#Page' + val + 'Label').show();
        $('#Page' + val + 'Link').hide();
        $('#pageViews').text(data.PageViews);
        $('#visits').text(data.Visits);
        $('#downloads').text(data.Downloads);
      },
      'json'
    );
    return false;
  });
});

Basically, that means we can harvest those statistics as well, by POSTing to http://runnet.codeplex.com/stats/getActivity and supplying a period argument. Some more digging around reveals that the argument can be 7, 30 or -1 for 'All'.

Aggregating Codeplex statistics

The following code snippet is the heart of a console application that retrieves those statistics for a series of projects (the ones we host on Codeplex, of course):

  ''' <summary>
  ''' Download and show Codeplex statistics on a series of projects
  ''' </summary>
  Sub Main()
    ' Show header
    Console.WriteLine(
      String.Format(
        "{0,-40} {1,15} {2,15} {3,15}",
        "Project:",
        "Page views:",
        "Visits:",
        "Downloads:"
      )
    )

    ' Loop over projects
    For Each projectName As String In
    {
      "MOBZHash",
      "MOBZHunt",
      "MOBZKeys",
      "MOBZoom",
      "MOBZPing",
      "MOBZRuler",
      "MOBZync",
      "QuickCode",
      "RegFind",
      "RegName",
      "RunNET",
      "SeeThroughWindows",
      "ShellRunner"
    }
      ' Get activity information for a project
      ' Note: -1 is 'All'
      Dim actInfo As ActivityInfo = GetActivityForProject(projectName, -1)

      ' Show activity info
      Console.WriteLine(
        String.Format(
          "{0,-40} {1,15} {2,15} {3,15}",
          projectName,
          actInfo.pageViews,
          actInfo.visits,
          actInfo.downloads
        )
      )
    Next
  End Sub

The ActivityInfo class is simply defined as:

  ''' <summary>
  ''' Data returned from GetActivity
  ''' </summary>
  Private Class ActivityInfo
    Public pageViews As Integer
    Public visits As Integer
    Public downloads As Integer
  End Class

The code that does the heavy lifting is basically adapted from MSDN. It creates a POST web request to /stats/getActivity on the project's domain on Codeplex, adds a period argument to the POST data and reads the response into a string. That string is then deserialized into an ActivityInfo object using JSON.NET.

Here's the code:

  ''' <summary>
  ''' Get the activity information for a Codeplex project
  ''' </summary>
  ''' <param name="projectName">The name of the project (as in [projectname].codeplex.com)</param>
  ''' <param name="period">7, 30 or -1 ("All")</param>
  ''' <returns>A populated ActivityInfo object</returns>
  ''' <remarks>No error handling!</remarks>
  Private Function GetActivityForProject(
    projectName As String,
    period As Integer
  ) As ActivityInfo

    ' Set up a WebRequest
    Dim request As WebRequest = WebRequest.Create(
      String.Format("http://{0}.codeplex.com/stats/getActivity", projectName)
    )
    ' Use POST to supply the parameters
    request.Method = "POST"
    request.ContentType = "application/x-www-form-urlencoded"

    ' Set up the POST data
    Dim postData As String = String.Format("period={0}", period)

    ' Write it to the request stream in UTF8 format
    Dim byteArray As Byte() = Encoding.UTF8.GetBytes(postData)
    ' Set Content length BEFORE writing
    request.ContentLength = byteArray.Length

    Using dataStream As Stream = request.GetRequestStream()
      dataStream.Write(byteArray, 0, byteArray.Length)
    End Using

    ' Get and process the response
    Using response As WebResponse = request.GetResponse()
      Using dataStream As Stream = response.GetResponseStream()
        Using reader As New StreamReader(dataStream)
          Dim responseFromServer As String = reader.ReadToEnd()

          ' Return an ActivityInfo from the content of the response
          Return JsonConvert.DeserializeObject(Of ActivityInfo)(
            responseFromServer
          )
        End Using
      End Using
    End Using
  End Function

The output of the resulting console application (the complete VB.NET code can be found in Codeplex Stats) is:

Project:                               Page views:         Visits:      Downloads:
MOBZHash                                        25               6               3
MOBZHunt                                      2040            1197            1207
MOBZKeys                                        28               7               3
MOBZoom                                        428             210             267
MOBZPing                                       672             248             256
MOBZRuler                                      776             429             540
MOBZync                                       3558            2325            2808
QuickCode                                      477             175             145
RegFind                                        844             322             249
RegName                                       1036             569             731
RunNET                                         162              79              40
SeeThroughWindows                             9243            6127            6336
ShellRunner                                    535             144             182

Of course, the data presented here must still be imported into a database or a spreadsheet to be useful over time, but that's left as an exercise for the reader ;-)

As usual: enjoy!

PS: Since there is - to my knowledge - no public API to get these statistics, the usual caveat applies: the interface with /stats/getActivity may change without notice, so any code that depends on it (i.e. all of the above) may break anytime.