Abstract

As the vast majority of network measures are defined for one-mode networks, two-mode networks often have to be projected onto one-mode networks to be analyzed. A number of issues arise in this transformation process, especially when analyzing ties among nodes’ contacts. For example, the values attained by the global and local clustering coefficients on projected random two-mode networks deviate from the expected values in corresponding classical one-mode networks. Moreover, both the local clustering coefficient and constraint (structural holes) are inversely associated to nodes’ two-mode degree. To overcome these issues, this paper proposes redefinitions of the clustering coefficients for two-mode networks.

Motivation

The clustering coefficients for one-mode networks are a measure of cohesion or group formation. These measures are defined around triplets (i.e., three nodes with at least two ties among them) and whether or not these triplets are closed (i.e., they form part of a triangle). Two-mode networks are often projected onto one-mode networks to be analysed. These networks often contain many more triangles than prototypical networks, and thus overestimates the level of clustering in a network. Methodological issues exist at a local level as well. Specifically, when calculating the local clustering coefficient (Watts and Strogatz, 1998) or the structural holes measure constraint (Burt, 1992) on projected two-mode networks, the measures are inversely correlated with nodes’ two-mode degree on a randomly tie reshuffled two-mode network (each node maintains their degree). Below is the average (a) local clustering coefficient and (b) constraint scores for nodes in a random version of the Scientific Collaboration Network (Newman, 2001) for various levels of two-mode degree.

As a result, a host of clustering measures for two-mode networks has been developed. For example, Robin and Alexander (2004) defined a coefficient as the number of four-cycles divided by the number of three-paths. Four-cycles in two-mode networks are the smallest possible cycle (like triangles are the smallest possible cycle in one-mode networks). However, this measure is distinctly different from the idea of triadic closure as the measure only include two primary nodes. In fact, a four-cycle is an indication of reinforcement or agreement between two-nodes and not cohesion or group formation.

The paper proposes redefinitions of the global and local clustering coefficients for two-mode networks. The measures are defined around 4-paths or triplets of primary nodes in two-mode networks. Specifically, the global coefficient is defined as the number of 4-paths that are closed divided by the total number, while the local is similar but focused on 4-paths centred on the focal node. For more details, see the paper (Social Networks; arXiv) or the tnet documentation (tnet » Two-mode Networks » Clustering).

Want to test it with your data?

The clustering_tm and clustering_local_tm-functions in tnet allows you to calculate the global and local clustering coefficients for two-mode networks (both binary and weighted) on your own dataset.

# Load tnet
library(tnet)

# Load a sample network (Figure 3A of the paper)
net <- rbind(
c(1,1),
c(1,2),
c(2,1),
c(2,3),
c(3,2),
c(3,3),
c(4,3))

# Calculate global clustering coefficient
clustering_tm(net)

# Calculate local clustering coefficient
clustering_local_tm(net)

References

Burt, R.S., 1992. Structural holes. Harvard University Press, Cambridge, MA.

Newman, M. E. J., 2001. Scientific collaboration networks. II. Shortest paths, weighted networks, and centrality. Physical Review E 64, 016132.

Opsahl, T., 2013. Triadic closure in two-mode networks: Redefining the global and local clustering coefficients. Social Networks 35, doi: 10.1016/j.socnet.2011.07.001.

Robins, G., Alexander, M., 2004. Small worlds among interlocking Directors: Network structure and distance in bipartite graphs. Computational and Mathematical Organization Theory 10 (1), 69–94.

Watts, D. J., Strogatz, S. H., 1998. Collective dynamics of “small-world” networks. Nature 393, 440-442.

1. Reprogramme everything in (and learn) C++
2. Get more resources

While the first solution might be the most appropriate one for repetitive tasks or production code, the second one might be quicker and easier for data scientists doing one-off analyses. It is possible to get more resources without buying a new server or more memory chips by using cloud computing. In essence, cloud computing allows analysts to rent resources when they need them. Amazon is one provider with the Elastic Compute Cloud (EC2). For example, it is possible to rent a server with 8 cores and 68.4 GB of memory for $2 per hour instead of buying a $5,000+ server.

The downside to using cloud computing is security. When I first started using R on Amazon EC2, I followed the instructions Bioconductor in the cloud (Thanks guys for maintaining an updated AMI and a great tutorial!). This tutorial shows how to set up an account, creating public/private keys, changing the firewall, launching a machine image with the latest version of R pre-installed, connecting to the command line using Secure Shell or SSH, using R through the command line interface, and using R through a web browser (RStudio). While the command line interface is secure using SSH and the private key, the web interface is not secure (standard username, password, and port as well as non-encrypted traffic). This means that anyone who knows the hostname or ip address could login to an R session. In fact, a port scan of the standard port across the range of ip addresses could allow a hacker to detect vulnerable servers and get access to their computing resources and data. In the rest of this post, I will borrow from Bioconductor in the cloud but suggest points to increase the security of using their machine image on Amazon EC2.

This tutorial is basic and highlights all the steps needed to get up and running. It is written using Windows and Putty, but should be applicable to people using other software with a bit of fudging. Before you start, please download PuTTY and PuTTYgen and save them in a convenient location. Note: These programmes do not need to be installed.

Getting to the Management Console

The first thing to do is to set up an Amazon account (yes, the same one as you buy books with), and register for Amazon Web Services (AWS). Then you need to go to the AWS Management Console. The link will be on the very top of the page when you are signed in with the Amazon id. The Management Console controls a number of AWS products, so you want to go to Elastic Compute Cloud by clicking on “Amazon EC2″ on the horizontal menu.

Amazon Web Service’s Management Console: The EC2 Dashboard

Public/private key pair

The first thing to do in the Management Console is to create a public/private key pair. This is important because the private key will be the “username and password” when accessing the server. This is done by clicking on “0 Key Pairs” under “My Resources” on the right, and then “Create Key Pair”. You need to give it an arbitrary name and then save the file with that-name.pem somewhere safe on your computer.

Unlike other SSH programmes, PuTTY cannot read pem-files directly. They must be converted to ppk-files. This can be done using PuTTYgen. When running the programme, click on the Load button in the middle of the screen (do not use the top menu!). Change the file-type drop-down menu from “PuTTY Private Key Files (*.ppk)” to “All Files (*.*), and select the pem-file with the private key downloaded from the Management Console. Then, click on the “Save private key” button to save the private key as a ppk-file. You do not need to password protect the file if you store it in a location only you have access to.

Firewall settings

Communication to a server occurs through ports. A firewall is a mechanism for opening some ports and closing others. A part of maintain a secure server is to only open the ones you need. The Bioconductor in the cloud-guide suggests that you open ports 22 and 8787. Port 22 is used to connect to the command line of the server using SSH. Port 8787 is the web interface of RStudio. While SSH traffic is encrypted, http traffic towards port 8787 is not. As such, this is a potential security vulnerability. I suggest that you only open port 22. Later on in this post, I will show how you can reach the web interface securely over port 22.

The firewall protecting servers on Amazon EC2 is controlled through the EC2 Dashboard’s Security Groups. A security group is a collection of instructions or rules. By default, there should be one security group called default. To see the details of this group, click on “1 Security Group” on the EC2 Dashboard and then click on “default”. The rules are listed under the “Inbound”-tab. If “22 (SSH)” is not listed, you need to open it. This is done by selecting SSH from the drop-down menu, clicking “+Add Rule”, and then clicking “Apply Rule Changes”. The servers with the default security settings will now be reachable on port 22.

Amazon EC2 Default Firewall Settings with SSH

Running a server

Now you have completed all the one-off set-up tasks, and you are ready to launch a server or instance. By clicking on “Instances” on the left-side, you should see the instances running as well as being able to start new one. Click on “Launch Instance” to get started, and select “Launch Classic Wizard”. There are five parts to this process:

1: Choose an AMI

The first question you are asked is which kind of software system or machine image (AMI) you want. There are a number of standard ones, but to save time and many lines of code, I will show you how to make use of Bioconductor in the cloud‘s 64-bit Linux system with the latest version of R installed. To load this, select the Community AMIs-tab, enter ami-b5a079dc in the search box (R-2.15; check their website for a new AMI id when a new version of R is released), and then click the Select-button.

2: Instance Details

The next question you are asked is the resources you would like, and where you would like the server to be located (note that prices vary based on location with “US East (Virginia)” often being the cheapest). Please refer to Amazon’s current pricing table. At the time of writing, the Hi-Memory On-Demand Instances (Linux) cost the following:

By clicking continue, you will be offered a number of more advanced options. The default values are ok. On the third screen, you are asked to give the instance a name (e.g., R-server).

3: Create Key Pair

You should already have completed this part, so you should see the arbitrary name chosen in a drop-down box and be able to just click Continue.

4: Configure Firewall

We have also already completed this step, so make sure the default group is selected and click Continue.

5: Review

On the final page, you are able to review all the settings. Below is an example of a 68.4 memory instance.

Launching an AMI

After hitting launch, a server will be allocated and the AMI will be loaded onto it. When it is complete, the status light will turn green and state “running”. Do note that you are being charged from this moment. See the final section for information on how to stop being charged.

Connecting to the command line using SSH

To control the server, you need to use SSH. The first thing we need to find out is the address of the server. This information is found by click on a running instance in the Instance-page of the EC2 Dashboard under “Public DNS”. An address will be similar to “ec2-184-72-187-196.compute-1.amazonaws.com”. Write this address down, or more easily, copy it to your clipboard. I will use this example address in the rest of the post, remember to change it to the address of your instance!

In this post, I am showing how PuTTY can be used for this; however, there are a number of other programmes out there that does the same thing. In PuTTY, we need to enter the address of the server and load the ppk-file created earlier with the private key. The screenshots below show the server’s address entered under Session and the private key loaded under SSH > Auth.

PuTTY with private key as authentication

By clicking on Open, PuTTY connects to the server. The first time you connect to a server, you will have to accept the public key. You can check the finger print against the one listed on the Key Pairs-page of the EC2 Dashboard. When asked “login as:”, simply enter root to get full privileges on the server.

Running R and installing tnet using SSH

When you have access to the command line, you can start R by simply typing R and hitting enter. This version of R comes with the Bioconductor-packages. To install the latest version of tnet, you need to type install.packages("tnet")

After downloading and compiling tnet and its dependencies, you can load tnet by typing library(tnet)

Connecting securely to RStudio Server

There are certain limitations to using the command line interface with R. First, it does not allow for graphical representations. Second, it is more cumbersome than the standard R for Windows GUI. To overcome these limitations, RStudio Server can be used. This software is a nice GUI for Linux servers running R, and is pre-installed on the Bioconductor AMI. If you opened port 8787 that the Bioconductor in the cloud-tutorial suggests, you could reach this interface by typing http://ec2-184-72-187-196.compute-1.amazonaws.com:8787 in a web browser (remember to replace the address with the one of your instance). However, as mentioned above, this leaves a large security hole open and allows others to “borrow” the resources you are paying for as well as being able to steal your data.

It is possible to communicate securely with the RStudio using an SSH tunnel. An SSH tunnel is an encrypted wrapper for other internet traffic. It is possibly best described using a diagram:

Using an SSH tunnel to encrypt the connection between a web browser and RStudio server

In a standard connection, the web browser connects directly to the RStudio Server on port 8787 (e.g., http://ec2-184-72-187-196.compute-1.amazonaws.com:8787). This traffic can be intercepted. Conversely, when using an SSH tunnel, the web browser connects to PuTTY, which encrypts the traffic and sends it to the SSH server, which decrypts it and sends it to RStudio Server. By not opening port 8787 in the Firewall, RStudio Server is only available to people logged on to the server.

To configure PuTTY to run an SSH tunnel, you need to follow the instructions for connecting to the command line. Additionally, you need to enter the following details under SSH > Tunnels:

• Source port: 8787
• Destination: localhost:8787

Do remember to click “Add”. The panel should look similar to this:

Setting up an SSH tunnel in PuTTY for RStudio Server

When you then connect to the instance (click Open and login), the SSH tunnel will be active. Congratulations: You can then open a web browser and type http://localhost:8787 to securely connect to the instance. The default username and password are unbuntu and bioc. In RStudio, you can install tnet by selecting it from the “Packages”-tab in the lower-right panel.

The login screen of RStudio Server. The default username and password are unbuntu and bioc.

A second less-secure alternative

By looking at the length of this post, I do realise that there are quite a few steps to achieve a secure http connection with an Amazon EC2 instance. Although the above solution ensures that it is not possible to eavesdrop on the traffic between your computer and the EC2 instance, there is a simpler trick that should stop most people trying to log into your session: change the default password. You still need to connect to the command line using SSH. When you are there, you should write passwd ubuntu to be prompted to enter a new password. Note that this procedure would require you to open port 8787 in addition to port 22 in the firewall (instead of selecting SSH from the drop-down menu, select “Custom TCP rule” and enter 8787 in the port range). Having said that, I do strongly encourage taking the extra step and using an SSH tunnel to ensure that your data and resources are safe.

Stopping and Terminating

As a final note, it is important to stop instances when you are done with them. Otherwise, you will continue to be charged! This is done by selecting an instance on the Instance-page of the EC2 Dashboard, and selecting Stop or Terminate from the Instance Actions drop-down box. Stop means that the server will be shut-down, but all the data and programmes on it will be saved on the Amazon Elastic Block Store (EBS; not free but quite cheap). A stopped instance can easily be restarted by choosing it and selecting Start from the Instance Actions drop-down box. Conversely, Terminate stops an instance without saving it to EBS. It is not possible to restart a terminated instance.

Alaska is a sparsely populated, isolated region with a disproportionately large, for its population size, number of airports. Most Alaskan airports have connections only to other Alaskan airports. This fact makes sense geographically. However, distance-wise, it also would make sense for some Alaskan airports to be connected to airports in Canada’s Northern Territories. These connections are, however, absent. Instead, a few Alaskan airports, singularly Anchorage, are connected to the continental U.S. The reason is clear: the Alaskan population needs to be connected to the political centers, which are located in the continental U.S., whereas there are political constraints making it difficult to have connections to cities in Canada, even to ones that are close geographically ([Guimera and Amaral, 2004]). It is now obvious why Anchorage’s centrality is so large. Indeed, the existence of nodes with anomalous centrality is related to the existence of regions with a high density of airports but few connections to the outside. The degree-betweenness anomaly is therefore ultimately related to the existence of communities in the network.

While many researchers and practitioners highlight this finding, I do not believe it is completely accurate. There are two reasons for this:

Issue 1: Binary ties

Admittedly this might be a personal bias as most of my work has been on weighted networks. Without going into much detail in this blog post, I actually strongly believe that if you assign the same importance to the connection between London Heathrow and New York’s JFK as you do to the connection between Pack Creek Airport and Sitka Harbor Sea Plane Base in Alaska (map), then there is a potential for measurement error. The table below lists the top ten airports in terms of betweenness when analyzing the binary and weighted (by passengers) versions of the Bureau of Transportation Statistics (BTS) Transtats data (Brandes, 2001). The code to replicate these results can be found at the end of this page.

This table demonstrates that Anchorage has twice the betweenness of the runner-up, Fairbanks Alaska, in the binary analysis. In the weighted analysis, Anchorage loses the first place to Seattle and Fairbanks moves to 10th place. It is also worth noticing that only US airports are in the top ten lists using both analyses, which leads me on to the second issues with using the BTS data: sample selection.

Issue 2: Sample selection

This issue affects all network studies, and something I have been interested in for a while. We define a population and analyse the connections among them. For example, I have analysed the scientific collaboration network based on the papers uploaded to the arXiv preprint server (e.g., Opsahl et al., 2008) with the full knowledge that there are many more scientific publications out there as well as other forms of collaboration and channels for knowledge flow among scientists, such as grant proposals and conference attendance. By simply restricting ourselves to data that is easy to collect (often stored in a central location / repository), the research is vulnerable to sample selection bias.

When it comes to airport networks, the Bureau of Transportation Statistics (BTS) Transtats data is straight forward to collect: Go here, select what you want (Origin, Destination), and click Download! However, there is a small note on another page explaining the dataset: “This table combines domestic and international market data reported by U.S. and foreign air carriers, and contains market data by carrier, origin and destination, and service class for enplaned passengers, freight, and mail. For a uniform end date for the combined databases, the last 3 months U.S. carrier domestic data released in T-100 Domestic Market (U.S. Carriers Only) are not included. Flights with both origin and destination in a foreign country are not included.” It is the last line of this description that highlights the potential sample selection bias. While the data contain all US airports and all domestic flights, it only contains Non-US flights that leave or terminate at a US airport and the Non-US airports on the other end of these flights. As such, a section of the square adjacency matrix is missing (flights from Non-US to Non-US airports in the dataset) as well as the entire rows and columns for airports without flights to the US. To exemplify this bias, I have plotted the routes on a world map below.

The BTS airport network in 2010 where the line colour is based on the number of passengers. The code to replicate this image can be found at the end of this page. If you click on the image, a vector graphic version of it is available (pdf; 5.95mb).

While it is possible to see a concentration in the US on the above picture, the sample selection becomes much more apparent when highlighting Europe. In the picture to below, it is possible to see that no flights are between any pair of European cities nor any other point on the map. Would you have to transit at New York’s JFK to get from London to Barcelona? This gap highlights the need for looking for more complete data sources than the Bureau of Transportation Statistics when analysing airport networks.

The European part of the BTS airport network

Alternative data-source

There are a couple of authoritative databases with world-wide airline routes. However, most of them are proprietary as they have enormous business intelligence potential and, as a consequence, are difficult to collect. OAG Worldwide is one such database, and it should be noted that Guimera et al (2004) went through the hoops by getting this data, and therefore, had a much more complete view of the airport network than if they had used the BTS Transtats data. While I do not have access to such a database, Openflights.org is a crowdsourced alternative. Although using this data comes without any guarantee, it has the potential to showcase the limitations of the BTS Transtats data. As a first step, I mapped the data to ensure there were no obvious pockets of missing data.

The Openflight.org airport network where the line colour is based on the number of routes (accessed on August 12, 2011). The code to replicate this image can be found at the end of this page. If you click on the image, a vector graphic version of it is available (pdf; 5.25mb).

Conclusion 1: Anchorage is not the most important airport

As can be seen from this picture, there are no obvious areas without any form of airline traffic. To show how this data impacts on a betweenness analysis, I have computed betweenness on both the binary and weighted (by number of routes as the passenger numbers were not available) versions of the network. As can be seen in the table below, major airports located around the globe get the highest scores in these analyses instead of only US airports. Specifically, Anchorage is only the third most central in the binary analysis, and the 14th most central in the weighted analysis. As such, it is still an important airport in the networks, but maybe not the most important.

_
Conclusion 2: Finding the global superhub using a weighted approached

London Heathrow is the most central airport when considering both tie weights and the global airport network. And this, unlike Anchorage, is not a surprising finding as it is the airport with most international passengers (Airports Council International, 2011).

To further investigate the effects on the ranking when considering tie weights in the global airport networks, I considered the change in ranking of the two airports ranked first in the binary and weighted analyses, Frankfurt and London Heathrow. Frankfurt went from having the highest betweenness in the binary analysis to only having 18th highest betweenness in the weighted analysis. Conversely, London Heathrow went from having the seventh highest to the highest betweenness score. To look into this cross-over of rankings, I compared the degree (number of airports with direct flights) and strength (number of direct routes) from these two airports:

This table shows that Frankfurt has direct flights to 51% more airports than London Heathrow, but only 21% more routes. The variation in tie weights can be further investigated by looking at the weight distribution. While there are only four airports with four direct routes from Frankfurt, there are nine airports with four or five direct routes from London Heathrow.

Moreover, by looking at which airports have the strong ties (i.e., with tie weights greater or equal to 4) with Frankfurt and London Heathrow, it is possible to see that the geographical distribution is strikingly different. Frankfurt has four direct routes to Antalya (Turkey), Madrid (Spain), Mallorca (Spain), and Vienna (Austria), which are 5,597 kilometres long (average: 1,399km). Conversely, London Heathrow has five routes to Delhi (India), Dubai (UAE), Hong Kong (China), Los Angeles (LAX, USA), and New York City (JFK, USA) and four routes to Bangkok (Thailand), Mumbai (India), Boston (USA), and Miami (USA), which are 65,376 kilometres long (average 7,264km). By having strong ties to geographically distant instead of close airports, London Heathrow acts as a intercontinental hub instead of a continental hub. Additionally, the airports with strong ties to London Heathrow have high betweenness, and therfore, act as hubs in their respective regions. As such, London Heathrow can be seen as the global hub of the world-wide airport network.

References

Airports Council International, 2011. Year to date International Passenger Traffic, Apr-2011, accessed August 12, 2011.

Brandes, U., 2001. A Faster Algorithm for Betweenness Centrality. Journal of Mathematical Sociology 25, 163-177.

Bureau of Transportation Statistics, 2011. Air Carrier Statistics (Form 41 Traffic): T-100 Market (All Carriers), accessed August 12, 2011.

Guimera, R., Amaral, L. A. N., 2004. Modeling the world-wide airport network. The European Physical Journal B 38, 381–385.

Guimera, R., Mossa, S., Turtschi, A., Amaral, L. A. N., 2004. The worldwide air transportation network: Anomalous centrality, community structure, and cities’ global roles. Proceedings of the National Academy of Sciences 102(22), 7794-7799.

Openflights.org, 2011. Airport, airline and route data, accessed August 12, 2011.

Opsahl, T., Agneessens, F., Skvoretz, J., 2010. Node centrality in weighted networks: Generalizing degree and shortest paths. Social Networks 32 (3), 245-251.

Opsahl, T., Colizza, V., Panzarasa, P., Ramasco, J. J., 2008. Prominence and control: The weighted rich-club effect. Physical Review Letters 101 (168702).

Code used to create the results in this blog post

Below is the code to redo the analysis in this post. You need to have the R-packages geosphere, maps, and tnet installed before to run the code. You also need to download the Bureau of Transportation Statistics (BTS) Transtats data. Please see notes in the code.

###################################
## US Airport network (BTS data) ##
###################################

# Load BTS Transtats data
# Downloaded from http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=292 using
# Filters: Geography=all; Year=2010; Months=all
# Columns: Passengers, Origin, OriginCountryName, Dest, DestCountryName
BTS <- read.csv("data/344989982_T_T100_MARKET_ALL_CARRIER.csv", header=TRUE, stringsAsFactors=FALSE)
BTS <- BTS[,c("ORIGIN", "ORIGIN_COUNTRY_NAME", "DEST", "DEST_COUNTRY_NAME", "PASSENGERS")]

# Load airport information (incl. geolocations)
# Downloaded from http://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=288 (select all columns)
BTSairports <- read.csv("data/344990073_T_MASTER_CORD.csv", stringsAsFactors=FALSE)

# Replace airport codes with id numbers (net1)
net1 <- BTS
net1.labels <- unique(c(net1[,"ORIGIN"], net1[,"DEST"]))
net1.labels <- net1.labels[order(net1.labels)]
net1[,"ORIGIN"] <- factor(x=net1[,"ORIGIN"], levels=net1.labels)
net1[,"DEST"]   <- factor(x=net1[,"DEST"], levels=net1.labels)
net1 <- data.frame(i=as.integer(net1[,"ORIGIN"]), j=as.integer(net1[,"DEST"]), w=net1[,"PASSENGERS"])

# Add up duplicated entries (multiple routes)
net1 <- net1[order(net1[,"i"], net1[,"j"]),]
index <- !duplicated(net1[,c("i","j")])
net1 <- data.frame(net1[index,c("i","j")], w=tapply(net1[,"w"], cumsum(index), sum))

# Take out routes with no passengers (cargo)
net1 <- net1[net1[,"w"]>0,]

# Take out routes from an airport to itself
net1 <- net1[net1[,"i"]!=net1[,"j"],]

# Load tnet and the network as a tnet object
library(tnet)
net1 <- as.tnet(net1, type="weighted one-mode tnet")

# Calculate binary and weighted betweenness
tmp0 <- betweenness_w(net1, alpha=0)
tmp1 <- betweenness_w(net1, alpha=1)

# Create output object with top x airports
x <- 10
out <- data.frame(
  tmp0[order(-tmp0[,"betweenness"]),][1:x,],
  tmp1[order(-tmp1[,"betweenness"]),][1:x,])
dimnames(out)[[2]] <- c("BTS.bb.node", "BTS.bb.score", "BTS.wb.node", "BTS.wb.score")
BTSairports[BTSairports[,"TR_COUNTRY_NAME"]=="United States of America","TR_COUNTRY_NAME"]  <- "USA"
for(i in 1:x) {
  # Insert label of airport ID (binary)
  tmp2 <- net1.labels[as.integer(out[i,"BTS.bb.node"])][1]
  tmp2 <- BTSairports[BTSairports[,"AIRPORT"]==tmp2,][1,]
  out[i,"BTS.bb.node"] <- paste(tmp2["AIRPORT"], " (", tmp2["TR_CITY_NAME"], ", ", tmp2["TR_COUNTRY_NAME"], ")", sep="")
  # Insert label of airport ID (weighted)
  tmp2 <- net1.labels[as.integer(out[i,"BTS.wb.node"])][1]
  tmp2 <- BTSairports[BTSairports[,"AIRPORT"]==tmp2,][1,]
  out[i,"BTS.wb.node"] <- paste(tmp2["AIRPORT"], " (", tmp2["TR_CITY_NAME"], ", ", tmp2["TR_COUNTRY_NAME"], ")", sep="")
}


#########################
## Plot the US network ##
#########################

# Based on FlowingData's blog post (http://flowingdata.com/2011/05/11/how-to-map-connections-with-great-circles/) Thanks Nathan! 
# Load required packages (type ?install.packages if you get an error)
library(maps)
library(geosphere)

# Symmetrise the network to get the correct tie weight for visualisation as the two directed ties are plotted on top of each other
net1s <- as.data.frame(symmetrise_w(net1, method="SUM"))
net1s <- net1s[net1s[,"i"]<net1s[,"j"],]

# Put labels back in summed up network (i.e, no duplicates) 
net1s[,"i"] <- net1.labels[net1s[,"i"]]
net1s[,"j"] <- net1.labels[net1s[,"j"]]

# Sort data so that weak ties are plotted first
net1s <- net1s[order(net1s[,"w"]),]

# Set up world map and colors for lines
pdf("airport_BTS_plot.pdf", width=11, height=7)
map("world", col="#eeeeee", fill=TRUE, bg="white", lwd=0.05)
pal <- colorRampPalette(c("#cccccc", "black"))
colors <- pal(length(unique(net1s[,"w"])))
colors <- rep(colors, times=as.integer(table(net1s[,"w"])))

# Plot ties
for(i in 1:nrow(net1s)) {
  # Get longitude and latitude of the two airports
  tmp1 <- BTSairports[BTSairports["AIRPORT"]==net1s[i,"i"],c("LONGITUDE","LATITUDE")][1,]
  tmp2 <- BTSairports[BTSairports["AIRPORT"]==net1s[i,"j"],c("LONGITUDE","LATITUDE")][1,]
  # Get the geographical distance to see how many points on the Great Circle to plot
  tmp3 <- 10*ceiling(as.numeric(log(3963.1 * acos((sin(tmp1[2]/(180/pi))*sin(tmp2[2]/(180/pi)))+(cos(tmp1[2]/(180/pi))*cos(tmp2[2]/(180/pi))*cos(tmp1[1]/(180/pi)-tmp2[1]/(180/pi)))))))
  # Line coordinates
  inter <- gcIntermediate(tmp1, tmp2, n=round(tmp3), addStartEnd=TRUE, breakAtDateLine=TRUE)
  # Plot one line if the line does not cross the date line; two if so
  if(is.matrix(inter)) {
    lines(inter, col=colors[i], lwd=0.6)
  } else {
    for(j in 1:length(inter))
      lines(inter[[j]], col=colors[i], lwd=0.6)
  }
}
dev.off()


##########################
## Openflights.org data ##
##########################

# Download airport geolocations from openflights.org/data.html, and set column headings
# "http://openflights.svn.sourceforge.net/viewvc/openflights/openflights/data/airports.dat"
OFairports <- read.csv("data/airports.dat", header=FALSE, stringsAsFactors=FALSE)
dimnames(OFairports)[[2]] <- c("Airport ID", "Name", "City", "Country", "IATA/FAA", "ICAO", "Latitude", "Longitude", "Altitude", "Timezone", "DST")

# Download routes from openflights.org/data.html
# "http://openflights.svn.sourceforge.net/viewvc/openflights/openflights/data/routes.dat"
OF <- read.csv("data/routes.dat", header=FALSE, stringsAsFactors=FALSE)
dimnames(OF)[[2]] <- c("Airline", "Airline ID", "Source airport", "Source airport ID", "Destination airport", "Destination airport ID", "Codeshare", "Stops", "Equipment")

# Remove code-shares as these are duplicated entries
net2 <- OF[OF[,"Codeshare"]=="",c("Source airport ID", "Destination airport ID")]

# Take out routes from an airport to itself and the missing cases (~1%)
net2 <- net2[net2[,"Source airport ID"]!=net2[,"Destination airport ID"],]
net2 <- net2[net2[,"Source airport ID"]!="\\N",]
net2 <- net2[net2[,"Destination airport ID"]!="\\N",]

# As passengers per route is not available, create a weighted network with the weight equal to number of routes
net2 <- data.frame(i=as.integer(net2[,"Source airport ID"]), j=as.integer(net2[,"Destination airport ID"]))
net2 <- shrink_to_weighted_network(net2)


##################################
## Plot the OpenFlights network ##
##################################

# Symmetrise data for visualisation
net2s <- as.data.frame(symmetrise_w(net2, method="SUM"))
net2s <- net2s[net2s[,"i"]<net2s[,"j"],]

# Sort data so that weak ties are plotted first
net2s <- net2s[order(net2s[,"w"]),]

# Set up world map and colors for lines
pdf("airport_OF_plot.pdf", width=11, height=7)
map("world", col="#eeeeee", fill=TRUE, bg="white", lwd=0.05)
pal <- colorRampPalette(c("#cccccc", "black"))
colors <- pal(length(unique(net2s[,"w"])))
colors <- rep(colors, times=as.integer(table(net2s[,"w"])))

# Plot ties
for(i in 1:nrow(net2s)) {
  # Get longitude and latitude of the two airports
  tmp1 <- as.numeric(OFairports[OFairports["Airport ID"]==net2s[i,"i"],c("Longitude","Latitude")][1,])
  tmp2 <- as.numeric(OFairports[OFairports["Airport ID"]==net2s[i,"j"],c("Longitude","Latitude")][1,])
  # Get the geographical distance to see how many points on the Great Circle to plot
  tmp3 <- 10*ceiling(as.numeric(log(3963.1 * acos((sin(tmp1[2]/(180/pi))*sin(tmp2[2]/(180/pi)))+(cos(tmp1[2]/(180/pi))*cos(tmp2[2]/(180/pi))*cos(tmp1[1]/(180/pi)-tmp2[1]/(180/pi)))))))
  # Line coordinates
  inter <- gcIntermediate(tmp1, tmp2, n=round(tmp3), addStartEnd=TRUE, breakAtDateLine=TRUE)
  # Plot one line if the line does not cross the date line; two if so
  if(is.matrix(inter)) {
    lines(inter, col=colors[i], lwd=0.6)
  } else {
    for(j in 1:length(inter))
      lines(inter[[j]], col=colors[i], lwd=0.6)
  }
}
dev.off()


#####################################
## Analyse the OpenFlights network ##
#####################################

# Calculate binary and weighted betweenness (on the directed network, net2)
tmp0 <- betweenness_w(net2, alpha=0)
tmp1 <- betweenness_w(net2, alpha=1)

# Create output object with top x airports
out <- data.frame(out,
  tmp0[order(-tmp0[,"betweenness"]),][1:x,],
  tmp1[order(-tmp1[,"betweenness"]),][1:x,])
dimnames(out)[[2]][5:8] <- c("OF.bb.node", "OF.bb.score", "OF.wb.node", "OF.wb.score")
for(i in 1:x) {
  # Insert label of airport ID (binary)
  tmp2 <- OFairports[OFairports[,"Airport ID"]==out[i,"OF.bb.node"],]
  out[i,"OF.bb.node"] <- paste(tmp2["IATA/FAA"], " (", tmp2["City"], ", ", tmp2["Country"], ")", sep="")
  # Insert label of airport ID (weighted)
  tmp2 <- OFairports[OFairports[,"Airport ID"]==out[i,"OF.wb.node"],]
  out[i,"OF.wb.node"] <- paste(tmp2["IATA/FAA"], " (", tmp2["City"], ", ", tmp2["Country"], ")", sep="")
}


###########################
## Comparing FRA and LHR ##
###########################

# Get FRA and LHR's airport ids
ids <- sapply(c("FRA", "LHR"), function(a) OFairports[OFairports[,"IATA/FAA"]==a,"Airport ID"])
# Rank and Score of FRA
tmp1 <- as.data.frame(tmp1[order(-tmp1[,"betweenness"]),])
tmp1[tmp1[,"node"]==ids["FRA"],]
# Degree and Node strength
tmp3 <- degree_w(net2)
tmp3[ids,]
# Weight distribution
sapply(ids, function(a) table(net2[net2[,"i"]==a,3]))
# Airports with strong ties (w>=4)
tmp4 <- lapply(ids, function(a) data.frame(net2[net2[,"i"]==a & net2[,"w"]>=4,], label="", geo.dist=NaN, stringsAsFactors=FALSE))
# Insert labels
for(a in 1:2) {
  for(b in 1:nrow(tmp4[[a]])) {
    tmp2 <- OFairports[OFairports[,"Airport ID"]==tmp4[[a]][b,"j"],][1,]
    tmp4[[a]][b, "label"] <- paste(tmp2["IATA/FAA"], " (", tmp2["City"], ", ", tmp2["Country"], ")", sep="")
  }
}
# Geographical distance
for(a in 1:2) {
  tmp5 <- as.numeric(OFairports[OFairports["Airport ID"]==ids[a],c("Longitude","Latitude")][1,])
  for(b in 1:nrow(tmp4[[a]])) {
    tmp6 <- as.numeric(OFairports[OFairports["Airport ID"]==tmp4[[a]][b,"j"],c("Longitude","Latitude")][1,])
    tmp4[[a]][b, "geo.dist"] <- 6378.7 * acos((sin(tmp5[2]/(180/pi))*sin(tmp6[2]/(180/pi)))+(cos(tmp5[2]/(180/pi))*cos(tmp6[2]/(180/pi))*cos(tmp5[1]/(180/pi)-tmp6[1]/(180/pi))))
  }
}
sapply(1:2, function(a) mean(tmp4[[a]][,"geo.dist"]))
sapply(1:2, function(a) sum(tmp4[[a]][,"geo.dist"]))

where j represents all other nodes, N is the total number of nodes, and x is the adjacency matrix, in which the cell is defined as 1 if node i is connected to node j, and 0 otherwise.

Degree has generally been extended to the sum of weights when analysing weighted networks, and labeled node strength (Barrat et al., 2004). This measure can be formalised as follows:

where w is the weighted adjacency matrix, in which is greater than 0 if the node i is connected to node j, and the value represents the weight of the tie. This is equal to the definition of degree if the network is binary, i.e. each tie has a weight of 1. Conversely, in weighted networks, the outcomes of these two measures are different. Since node strength takes into consideration the weights of ties, this has been the preferred measure for analyzing weighted networks (e.g., Barrat et al., 2004; Opsahl et al., 2008).

Degree and Strength: Two nodes with the same node strength, but different number of ties.

where is a positive tuning parameter that controls the relative importance of the number of ties and the sum of ties. Specifically, there are two benchmark values (0 and 1), and if the parameter is set to either of these values, the existing measure is reproduced. If the parameter is set to the benchmark value of 0, the outcomes of the measure is solely based on the number of ties, and are equal to the ones found when applying Freeman’s (1978) measure to a binary version of a network where all the ties with a weight greater than 0 are set to present. Conversely, if the value of the parameter is 1, the outcomes of the measure is based on tie weights only, and are identical to the already proposed generalization of degree (Barrat et al., 2004). For other values of , alternative outcomes are attained, which are based on both the number of ties and tie weights. In particular, two ranges of values can be distinguished. First, a parameter set between 0 and 1 would positively value both the number of ties and tie weights. This implies that both increments in node degree and node strength will then increase the outcome. Second, if the value of the parameter is above 1, the measures would positively value tie strength and negatively value the number of ties. Nodes with on average stronger ties will get a higher score.

Variation in Tie Weights: Two nodes with the same scores using Freeman's (1978), Barrat et al.'s (2004), and Opsahl et al.'s (2010) degree measures.

By exponenting the tie weight instead of the average tie weight, the measure becomes sensitive to variation in tie weights. For example, node A and node B would get the following score using the various measures:

As it is possible to see from the above table, the new measure is closely linked to generalisation proposed by Opsahl et al. (2010); however, when the tie weights are different, the measure vary between the two nodes. Similarly as the other centrality measures using a tuning parameter, the tuning parameter in these measures control the relative importance of the number of ties and the sum of ties. In addition, it also controls whether variation in tie weights should be discounted or taken favourable. A parameter between 0 and 1 discounts, whereas a parameter above 1, increase the outcome of the measure when tie weights are different.

What to try it with your data?

Below is the code to calculate the proposed degree measure. You need to have the R-package tnet installed before to run the code.

# Load tnet
library(tnet)

# Load a function to calculate the new measures
degree2_w <- function (net, type="out", alpha = 1) {
    net <- as.tnet(net, type="weighted one-mode tnet")
    if (type == "in") {
        net <- data.frame(i = net[, 2], j = net[, 1], w = net[,3])
        net <- net[order(net[, "i"], net[, "j"]), ]
    }
    index <- cumsum(!duplicated(net[, 1]))
    k.list <- cbind(unique(net[, 1]), NaN, NaN, NaN)
    dimnames(k.list)[[2]] <- c("node", "degree", "output", "alpha")
    k.list[, "degree"] <- tapply(net[, "w"], index, length)
    k.list[, "output"] <- tapply(net[, "w"], index, sum)
    net[,"w"] <- net[,"w"]^alpha
    k.list[, "alpha"] <- tapply(net[, "w"], index, sum)
    if (max(net[, c("i", "j")]) != nrow(k.list)) {
        k.list <- rbind(k.list, cbind(1:max(net[, c("i", "j")]), 0, 0, 0))
        k.list <- k.list[order(k.list[, "node"]), ]
        k.list <- k.list[!duplicated(k.list[, "node"]), ]
    }
    return(k.list)
}

# Load a sample network
net <- cbind(
i=c(1,1,2,2),
j=c(2,3,1,3),
w=c(2,2,1,3))

# Calculate the measures
degree_w(net, measure=c("degree","output","alpha"), alpha=1.5)
degree_w(net, measure=c("degree","output","alpha"), alpha=0.5)
degree2_w(net, alpha=0.5)
degree2_w(net, alpha=1.5)

References

Barrat, A., Barthelemy, M., Pastor-Satorras, R., Vespignani, A., 2004. The architecture of complex weighted networks. Proceedings of the National Academy of Sciences 101 (11), 3747-3752.

Freeman, L. C., 1978. Centrality in social networks: Conceptual clarification. Social Networks 1, 215-239.

Opsahl, T., Agneessens, F., Skvoretz, J. (2010). Node centrality in weighted networks: Generalizing degree and shortest paths. Social Networks 32, 245-251.

Opsahl, T., Colizza, V., Panzarasa, P., Ramasco, J. J., 2008. Prominence and control: The weighted rich-club effect. Physical Review Letters 101 (168702).

Wasserman, S., Faust, K., 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, New York, NY.

The main foundation of the paper is a gender representation law that required all public limited companies to compose their boards with at least 40% of each gender by January 2008. The paper attempted to stike a balance between the urgency of studying the gender represention law and the amount of data available (the analysis relied on data from August 2009). The supporting website tries to alleviate this tension by providing up-to-date data for the analysis conducted in the paper.

The paper gathered news coverage in the major Norwegian newspapers. Online versions are available through e24.no and dn.no and Der Spiegel.

Abstract

Governments have implemented various affirmative action policies to address vertical sex segregation in organizations. A gender representation law was introduced in Norway, which required public limited companies’ boards to have at least 40 percent representation of each sex by 2008. This law acted as an external shock, and this paper aims to explore its effects. In particular, it explores the gender bias, the emergence and sex of prominent directors, and directors’ social capital. We utilize data from May 2002 to August 2009 to analyze these aspects. The implied intention of the law was to create a larger pool of women acting as directors on boards, and the law has had the effect of increasing the representation of women on boards. However, it has also created a small elite of women directors who rank among the top on a number of proxies of influence.

Abstract

Ties often have a strength naturally associated with them that differentiate them from each other. Tie strength has been operationalized as weights. A few network measures have been proposed for weighted networks, including three common measures of node centrality: degree, closeness, and betweenness. However, these generalizations have solely focused on tie weights, and not on the number of ties, which was the central component of the original measures. This paper proposes generalizations that combine both these aspects. We illustrate the benefits of this approach by applying one of them to Freeman’s EIES dataset.

Motivation

Ego networks of Phipps Arabie (A), John Boyd (B), and Maureen Hallinan (C) from Freeman's third EIES network. The width of a tie corresponds to the number of messages sent from the focal node to their contacts. Adopted from the paper.

The three measures have already been generalised to weighted networks. Barrat et al. (2004) generalised degree to weighted networks by taking the sum of weights instead of the number ties, while Newman (2001) and Brandes (2001) utilised Dijkstra’s (1959) algorithm of shortest paths for generalising closeness and betweenness to weighted networks, respectiviely. Dijkstra’s algorithm defined the length of paths as the sum of cost (e.g., time in GPS calculations), which is generally only defined as the sum of the inversed tie weights. All these generalisations fail to take into account the main feature of the original measures formalised by Freeman (1978): the number of ties.

This limitation is highlighted for degree centrality by the three ego networks from Freeman’s third EIES network. The three nodes have roughly sent the same amount of messages; however, to a quite different number of others. If Freeman’s (1978) original measure was applied, the centrality score of the node in panel A is almost five times as high as the node in panel C attains. However, when using Barrat et al.’s generalisation, they get roughly the same score.

This articles proposes a new generation of node centrality measures for weighted networks. The second generation of measures takes into consideration both the weight of ties and the number of ties. The relative importance of these two aspects are controlled by a tuning parameter.

Want to test it with your data?

The degree_w, closeness_w, and betweenness_w-functions in tnet allows you to calculate the binary, weighted, and the measures that combine these two aspects on your own dataset.

For example, to calculate second generation node centrality measures (alpha = 0.5) on the sample network above, you can run the code below in R. The degree function easily calculates the binary and first generation measures as well; however, this is not the case for the closeness and betweenness-functions. If you would like the binary version, you can either use the dichotomise function or set alpha=0. If you would like the first generation weighted measures, you can set alpha=1 (default value).

# Load tnet
library(tnet)

# Load network
net <- cbind(
i=c(1,1,2,2,2,2,3,3,4,5,5,6),
j=c(2,3,1,3,4,5,1,2,2,2,6,5),
w=c(4,2,4,4,1,2,2,4,1,2,1,1))

# Calculate degree centrality (note that alpha is included in the list of measures)
degree_w(net, measure=c("degree", "output", "alpha"), alpha=0.5)

# Calculate closeness centrality
closeness_w(net, alpha=0.5)

# Calculate betweenness centrality
betweenness_w(net, alpha=0.5)

To test it on Freeman’s third EIES network from the datasets-page and recreate Table 3 of the paper, you can do the following:

# Load tnet
library(tnet)

# Load network
data(Freemans.EIES)
net <- Freemans.EIES.net.3.n32

# Calculate measures
tmp <- data.frame(
  Freemans.EIES.node.Name.n32,
  degree_w(net, measure=c("degree", "output", "alpha"), alpha=0.5), 
  degree_w(net, measure="alpha", alpha=1.5)[,"alpha"], stringsAsFactors=FALSE)
dimnames(tmp )[[2]] <- c("name", "node", "a00", "a10", "a05", "a15")
tmp <- tmp[,c("name","a00","a05","a10","a15")]

# Merge names and order table
out <- data.frame(
  seq.int(nrow(tmp)),
  tmp[order(-tmp[,"a00"], -tmp[,"a10"]),c("name", "a00")],
  tmp[order(-tmp[,"a05"], -tmp[,"a10"]),c("name", "a05")],
  tmp[order(-tmp[,"a10"], -tmp[,"a10"]),c("name", "a10")],
  tmp[order(-tmp[,"a15"], -tmp[,"a10"]),c("name", "a15")])
dimnames(out)[[2]] <- c("Rank",
  "a00.name","a00",
  "a05.name","a05",
  "a10.name","a10",
  "a15.name","a15")

# Display table
out

References

Brandes, U., 2001. A Faster Algorithm for Betweenness Centrality. Journal of Mathematical Sociology 25, 163-177.

Dijkstra, E. W., 1959. A note on two problems in connexion with graphs. Numerische Mathematik 1, 269-271.

Freeman, L. C., 1978. Centrality in social networks: Conceptual clarification. Social Networks 1, 215-239.

Newman, M. E. J., 2001. Scientific collaboration networks. II. Shortest paths, weighted networks, and centrality. Physical Review E 64, 016132.

Opsahl, T., Agneessens, F., Skvoretz, J. (2010). Node centrality in weighted networks: Generalizing degree and shortest paths. Social Networks 32, 245-251.

This network gives a concrete example of the closeness measure. The distance between node G and node H is infinite as a direct or indirect path does not exist between them (i.e., they belong to separate components). As long as at least one node is unreachable by the others, the sum of distances to all other nodes is infinite. As a consequence, researchers have limited the closeness measure to the largest component of nodes (i.e., measured intra-component). The distance matrix for the nodes in the sample network is:

Although the intra-component closeness scores are not infinite for all the nodes in the network, it would be inaccurate to use them as a closeness measure. This is due to the fact that the sum of distances would contain different number of paths (e.g., there are two distance from node H to other nodes in its component, while there are six distances from node G to other nodes in its component). In fact, nodes in smaller components would generally be seen as being closer to others than nodes in larger components. Thus, researchers has focused solely on the largest component. However, this leads to a number of methodological issues, including sample selection.

To develop this measure, I went back to the original equation:

where is the focal node, is another node in the network, and is the shortest distance between these two nodes. In this equation, the distances are inversed after they have been summed, and when summing an infinite number, the outcome is infinite. To overcome this issue while staying consistent with the existing measure of closeness, I took advantage of the fact that the limit of a number divided by infinity is zero. Although infinity is not an exact number, the inverse of a very high number is very close to 0. In fact, 0 is returned if you enter 1/Inf in the statistical programme R. By taking advantage of this feature, it is possible to rewrite the closeness equation as the sum of inversed distances to all other nodes instead of the inversed of the sum of distances to all other nodes. The equation would then be:

To exemplify this change, for the example network above, the inversed distances and closeness scores are:

As can be seen from this table, a closeness score is attained for all nodes taking into consideration an equal number of distances for each node irrespective of the size of the nodes’ component. Moreover, nodes belonging to a larger component generally attains a higher score. This is deliberate as these nodes can reach a greater number of others than nodes in smaller components. The normalized scores are bound between 0 and 1. It is 0 if a node is an isolate, and 1 if a node is directly connected all others.

This measure can easily be extended to weighted networks by introducing Dijkstra’s (1959) algorithm as proposed in Average shortest distance in weighted networks.

References

Dijkstra, E. W., 1959. A note on two problems in connexion with graphs. Numerische Mathematik 1, 269-271.

Freeman, L. C., 1978. Centrality in social networks: Conceptual clarification. Social Networks 1, 215-239.

Opsahl, T., Agneessens, F., Skvoretz, J. (2010). Node centrality in weighted networks: Generalizing degree and shortest paths. Social Networks 32, 245-251.

Wasserman, S., Faust, K., 1994. Social Network Analysis: Methods and Applications. Cambridge University Press, New York, NY.

What to try it with your data?

Below is the code to calculate the closeness measure on the sample network above.

# Load tnet
library(tnet)

# Load network 
# Node K is assigned node id 8 instead of 10 as isolates at the end of id sequences are not recorded in edgelists
net <- cbind(
  i=c(1,1,2,2,2,3,3,3,4,4,4,5,5,6,6,7,9,10,10,11),
  j=c(2,3,1,3,5,1,2,4,3,6,7,2,6,4,5,4,10,9,11,10),
  w=c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1))

# Calculate measures
closeness_w(net, gconly=FALSE)

“we measured users’ average out-strength (instrength) as the average number of messages sent to (received from) others (Opsahl, Colizza, Panzarasa,&Ramasco, 2008). We expected hubs to be weakly connected to others, based on the conjecture that all users are homogeneously limited by the same constraints of resources and time. In this case, having more contacts should reduce the amount of resources and time spent on each of them (Burt, 1992). We were surprised to find a positive and significant (p<0.001) Pearson’s pairwise correlation coefficient between average out-strength (in-strength) and out-degree (in-degree) of 0.28 (0.44). This signals that hubs spend more time and resources with each of their contacts than the less connected users." (excerpt from page 919).

The heterogeneity in average tie weight for users with different levels of gregariousness might indicate that node degree and node strength are not correlated. This post aims to test this for the online social network used in the paper and compare degree and strength distributions.

Given that this is a directed network, each analysis is conducted twice – once for outgoing ties and once for incoming ties. The simplest way to test the association between two variables is to calculate the Pearson pair-wise correlation coefficient . This coefficient tests the linear relationship between two variables, and ranges from -1 to 1. If it is equal to 1, then there is perfect correlation between the two-variables, whereas if it is -1, the two variables are opposites of each other. A value of 0 is attained if there is no linear relationship between the two variables. For out-degree and out-strength, the coefficient is 0.90, and for in-degree and in-strength, the coefficient is 0.89. This indicates that degree and strength is highly correlated with each other (Cohen, 1988).

Since high correlation coefficients were found, it might be interesting to plot the relationships to ensure that extreme values are not distorting the coefficient. The relationships between the two types of degree and strength are:

As it is possible to see from the above plots, there are a number of nodes with extremely high values of degree and strength. However, there are clear trajectories at low values of degree and strength, which might indicate that the outliers are not distorting the correlation coefficients. The fact that there are nodes with extremely high values of degree is not surprising given that power-law degree distributions with exponents of 0.89 and 1.005 were found in the paper:

Given the similarity between degree and strength, it would be interesting to test whether the strength distributions also follow a power-law distribution, and if so, if the exponent is similar to the ones for the degree-distributions:

The exponents of the strength distributions are 0.87 and 1.004. Although I expected some similarity between the degree distributions’ exponents (0.89 and 1.005) and the strength distributions’ exponents, the numerical similarity is striking.

References

Burt, R. S., 1992. Structural Holes: The Social Structure of Competition. Harvard University Press, Cambridge, MA.

Cohen, J., 1988. Statistical power analysis for the behavioral sciences (2nd edition). Hillsdale, NJ: Erlbaum.

Opsahl, T., Colizza, V., Panzarasa, P., Ramasco, J. J., 2008. Prominence and control: The weighted rich-club effect. Physical Review Letters 101 (168702). arXiv:0804.0417.

Panzarasa, P., Opsahl, T., Carley, K.M., 2009. Patterns and dynamics of users’ behavior and interaction: Network analysis of an online community. Journal of the American Society for Information Science and Technology 60 (5), 911-932, doi: 10.1002/asi.21015

What to try it with your data?

Below is the code to calculate the numbers and create the diagrams used in this post. If you also would like to calculate the power-law with exponential cut-off, then you should remove the # on line 41.

# Load tnet
library(tnet)

# Load network
data(OnlineSocialNetwork.n1899)

`script` <-
function(net){
  output <- list()
  # Calculate out/in-degree/strength
  k <- cbind(degree_w(net), degree_w(net, type="in"))
  dimnames(k)[[2]] <- c("node","ko","so","node2","ki","si")
  if(sum(k[,"node"] == k[,"node2"])!=nrow(k))
    stop("Node ids does not match")
  k <- k[,c("node","ko","ki","so","si")]
  output[[1]] <- k

  # Get pair-wise correlation coefficients
  corro <- cor.test(k[,"ko"], k[,"so"])
  corri <- cor.test(k[,"ki"], k[,"si"])
  cat(paste("Pair-wise correlation between degree and strength:\n Out: ", corro$estimate, " (p-value: ", corro$p.value, ")\n In:  ", corri$estimate, " (p-value: ", corri$p.value, ")\n Note: If p-value equal 0, p-value is less than 2.2e-16\n", sep=""))
  output[[2]] <- corro
  output[[3]] <- corri

  # Degree distributions
  cat("Degree distributions\n")
  looprange <- c("ko","so","ki","si")
  for(j in 1:length(looprange)) {
    i <- looprange[j]
    tmp <- table(k[,i])
    tmp <- tmp[which(rownames(tmp)!="0")]
    tmp <- tmp/(sum(tmp))
    tmp <- as.data.frame(cbind(k=as.numeric(rownames(tmp)), pk=tmp))
    plaw <- nls(pk ~ C*k^(-t), data=tmp, start=list(C=1, t=1))
    plaweco <- nls(pk ~ C*k^(-t)*exp(-k/K), data=tmp, start=list(C=1, t=1, K=30))
    cat(switch(i,
      "ko" = " Out-degree",
      "so" = " Out-strength",
      "ki" = " In-degree",
      "si" = " In-strength"))
    cat(paste("\n  Powerlaw:  pk =", plaw$call$formula[3], "\n   Coefficients:\n    Con =", coef(plaw)["C"], "\n    tau =", coef(plaw)["t"]))
    # cat(paste("\n  Powerlaw with exponential cut-off: pk ", plaweco$call$formula[3], "\n   Coefficients:\n    Con =", coef(plaweco)["C"], "\n    tau =", coef(plaweco)["t"], "\n    cut =", coef(plaweco)["K"]))
    cat("\n")
    output[[(length(output)+1)]] <- tmp
    output[[(length(output)+1)]] <- plaw
    output[[(length(output)+1)]] <- plaweco
  }
  cat(" Note: These regressions in the article were performed in Stata 9\n The value of the cut-off parameter varies slightly between R and Stata\n")
  return(output)
}
output <- script(OnlineSocialNetwork.n1899.net)
k <- output[[1]]
plot(k[,"ko"], k[,"so"], main="Outgoing ties", xlab="out-degree", ylab="out-strength")
plot(k[,"ki"], k[,"si"], main="Incoming ties", xlab="in-degree",  ylab="in-strength" )

plot(output[[4]][,1], output[[4]][,2], main="Out-degree distribution", xlab="out-degree", ylab="p(out-degree)", log="xy")
lines(output[[4]][,1], fitted(output[[5]]))

plot(output[[7]][,1], output[[7]][,2], main="Out-strength distribution", xlab="out-strength", ylab="p(out-strength)", log="xy")
lines(output[[7]][,1], fitted(output[[8]]))

plot(output[[10]][,1], output[[10]][,2], main="In-degree distribution", xlab="in-degree", ylab="p(in-degree)", log="xy")
lines(output[[10]][,1], fitted(output[[11]]))

plot(output[[13]][,1], output[[13]][,2], main="In-strength distribution", xlab="in-strength", ylab="p(in-strength)", log="xy")
lines(output[[13]][,1], fitted(output[[14]]))

Acknowledgements

The theme of this thesis is interdependence among elements. In fact, this thesis is not just a product of myself, but also of my interdependence with others. Without the support of a number of people, it would not have been possible to write. It is my pleasure to have the opportunity to express my gratitude to many of them here.

For my academic achievements, I would like to acknowledge the constant support from my supervisors. In particular, I thank Pietro Panzarasa for taking an active part of all the projects I have worked on. I have also had the pleasure to collaborate with people other than my supervisors. I worked with Vittoria Colizza and Jose J. Ramasco on the analysis and method presented in Chapter 2, Kathleen M. Carley on an empirical analysis of the online social network used throughout this thesis, and Martha J. Prevezer on a project related to knowledge transfer in emerging countries. In addition to these direct collaborations, I would also like to thank Filip Agneessens, Sinan Aral, Steve Borgatti, Ronald Burt, Mauro Faccioni Filho, Thomas Friemel, John Skvoretz, and Vanina Torlo for encouragement and helpful advice. In particular, I would like to thank Tom A. B. Snijders and Klaus Nielsen for insightful reading of this thesis and many productive remarks and suggestions. I have also received feedback on my work at a number of conferences and workshops. I would like to express my gratitude to the participants at these.

On a social note, I would like to thank John, Claudius, and my family for their continuing support. Without them I would have lost focus. My peers and the administrative staff have also been a great source of support. In particular, I would like to extend my acknowledgements to Mariusz Jarmuzek, Geraldine Marks, Roland Miller, Jenny Murphy, Cathrine Seierstad, Lorna Soar, Steven Telford, and Eshref Trushin.

Abstract

In recent years, researchers have investigated a growing number of weighted networks where ties are differentiated according to their strength or capacity. Yet, most network measures do not take weights into consideration, and thus do not fully capture the richness of the information contained in the data. In this paper, we focus on a measure originally defined for unweighted networks: the global clustering coefficient. We propose a generalization of this coefficient that retains the information encoded in the weights of ties. We then undertake a comparative assessment by applying the standard and generalized coefficients to a number of network datasets.

Motivation

In this sample network the binary clustering coefficient is 0.33 as a third of the triplets are closed by being part of a triangle. By looking at the weights, it is possible to see that the strongest ties are in part of the closed triplets. This is not reflected in the binary clustering coefficient.

By applying the proposed generalisation of the coefficient using the arithmetic mean method for defining triplet value, the clustering coefficient increases to 0.42. This increase of this coefficient from the binary coefficient is a reflection of the fact that the strongest ties are part of the closed triplets.

Want to test it with your data?

The clustering_w function in tnet allows you to test the generalised clustering coefficient on your own dataset.

For example, to test the clustering_w function on the sample network above, you can run the following code in R:

# Load tnet
library(tnet)

# Load network
net <- cbind(
i=c(1,1,2,2,2,2,3,3,4,5,5,6),
j=c(2,3,1,3,4,5,1,2,2,2,6,5),
w=c(4,2,4,4,1,2,2,4,1,2,1,1))

# Run function
clustering_w(net, measure=c("am", "gm", "ma", "mi"))

# The output is:
#       am        gm        ma        mi
#0.4166667 0.4361302 0.3750000 0.5000000

To test in on Freeman’s third EIES network from the datasets page, you can do the following:

# Load tnet
library(tnet)

# Load network
data(Freemans.EIES)

# Run function
clustering_w(Freemans.EIES.net.3.n32, measure=c("am", "gm", "ma", "mi"))

# The output is:
#0.7378310 0.7331536 0.7410959 0.7249982

Abstract

This research draws on longitudinal network data from an online community to examine patterns of users’ behavior and social interaction and infer the processes underpinning dynamics of system use. The online community represents a prototypical example of a complex evolving social network in which connections between users are established over time by online messages. We study the evolution of a variety of properties since the inception of the system, including how users create, reciprocate, and deepen relationships with one another, variations in users’ gregariousness and popularity, reachability and typical distances among users, and the degree of local redundancy in the system. Results indicate that the system is a “small world” characterized by the emergence, in its early stages, of a hub-dominated structure with highly heterogeneous users’ behavior. We investigate whether hubs are responsible for holding the system together and facilitating information flow, examine first-mover advantages underpinning users’ ability to rise to system prominence, and uncover gender differences in users’ gregariousness, popularity, and local redundancy. We discuss the implications of the results for research on system use and evolving social networks, and for a host of applications, including information diffusion, communities of practice, and the security and robustness of information systems.

Abstract

Complex systems are often characterized by large-scale hierarchical organizations. Whether the prominent elements, at the top of the hierarchy, share and control resources or avoid one another lies at the heart of the global organization and functioning. Inspired by network perspectives, we propose a new general framework for studying the tendency of prominent elements to form clubs with exclusive control over the majority of a system’s resources. We explore associations between prominence and control in the fields of transportation, scientific collaboration, and online communication.

UPDATE: I have suggested an additional null model in Weighted Rich-club Effect: A more appropriate null model for scientific collaboration networks.

Want to test it with your data?

See the Weighted Rich-club Effect in the tnet manual.