alt

By Jonathon Morgan

Last week we drew a map of Syria using D3. The next step is to shade each city in the country based on its relative level of violence and create a choropleth visualization. We'll retrieve our data from the CrisisNET API using Node.js, and do some simple data formatting to package that information so it's easy to work with in D3. Check out last week's post for background on how D3 uses projections to render geospatial data in your browser, or get up and running using this GeoJSON file and the following snippet.

// Size of the canvas on which the map will be rendered
var width = 1000,  
    height = 1100,
    // SVG element as a JavaScript object that we can manipulate later
    svg = d3.select("#map").append("svg")
      .attr("width", width)
      .attr("height", height);

// Normally you'd look this up. This point is in the middle of Syria
var center = [38.996815, 34.802075];

// Instantiate the projection object
var projection = d3.geo.conicConformal()  
    .center(center)
    .clipAngle(180)
    // Size of the map itself, you may want to play around with this in 
    // relation to your canvas size
    .scale(10000)
    // Center the map in the middle of the canvas
    .translate([width / 2, height / 2])
    .precision(.1);

var path = d3.geo.path().projection(projection);

d3.json("cities.json", function(err, data) {  
  $.each(data.features, function(i, feature) {
    svg.append("path")
      .datum(feature.geometry)
      .attr("class", "border")
      .attr("d", path);
  });
});

Now that we have a map we can add color to each path element in our svg. Just like div and other DOM elements you're familiar with, path nodes can be assigned classes and styled with CSS. The CSS classes we assign to the path elements will relate to the number of reports about the city associated with that path. For example we might have class=count-10 for a city that reported then violent incidents, and a corresponding CSS rule shading this city with a darker opacity than other, less violent regions .count-10 { fill-opacity: .9; }.

First, let's get some data. We need the number of reports in CrisisNET associated with each city in our cities.json file. That file is a little cumbersome to work with, so let's quickly create a .csv file containing only the data we need to make our API requests. Note that in the Node.js examples below I'm using a few third-party libraries: Underscore.js, requests, and fast-csv, which you'll need to grab with npm.

var _ = require('underscore')  
  , fs = require('fs');

var cityJSON = require('cities.json')  
  , cities = fs.createWriteStream('sy-cities.csv');

_(cityJSON.features).each(function(feature) {  
  var toAdd = [feature.properties.PCODE, feature.properties.NAME_EN];
  cities.write(toAdd + "\n");
});

We're pulling two properties from every feature, PCODE and NAME_EN, and storing those in the sy-cities.csv file. Node.js file streams are outside the scope of this tutorial, but here's a handy post if you're not familiar with how they work. Here's some sample output:

SY070500,Ariha  
SY070501,Ehsem  
SY070502,Mhambal  

The PCODE is useful because it's a unique id for each city that will be available to our application as our map is rendering. Therefore we can use that value when looking up the number of violent incidents reported about that city. More on that in a moment. First let's get incident counts for every city in our new sy-cities.csv file, and associate that incident count with a PCODE in a new syria-incidents.csv file. Technically you could do this lookup while your map is rending, but because there are hundreds of cities in Syria, and you'd need to make a separate request for each one, that'd take forever. Instead we can cache the results in a CSV and avoid this tedious wait every time the map renders.

var _ = require('underscore')  
  , csv = require('fast-csv')
  , fs = require('fs')
  , request = require('request');

var incidents = fs.createWriteStream("syria-incidents.csv");  
var cities = fs.createReadStream("sy-cities.csv");

// add column headers as the first line in the file  
incidents.write(["cityID", "cityName", "totalIncidents"] + "\n");

// create a stream object and define its behavior
var csvStream = csv()  
  // this event is fired for each row in the csv you're reading
  .on("record", function(row){

    // each row has the name of a city, which we will use to make
    // a request to the api
    var searchTerms = {

      // this is the city name
      text: row[1], 

      // just to keep the response small, because we don't need any of
      // the returned documents. The API returns a total count of 
      // all documents matching these parameters in its response
      limit: 10, 
      placeName: 'syria'
      sources: 'facebook,youtube',

      // social content can be messy, so only look for posts that
      // were identified as being relevant to crisis
      tags: 'conflict',
      apikey: YOUR-API-KEY
    };

    request.get(url, { qs: searchTerms }, function(err, resp) {

      // PCODE, city name, total reports
      var row = [data[0], data[1], resp.total];
      incidents.write(row + "\n");
    });
  })
  .on("end", function(){
    console.log("done");
  });

// pipe the csv stream to the cities stream. Don't worry if you're not 
// familiar with Node file streams
cities.pipe(csvStream);  

Our syria-incidents.csv file should now be something like this:

cityID,cityName,totalIncidents  
SY070502,Mhambal,0  
SY070300,Harim,0  
...
SY010000,Damascus,2864  
SY100203,Qadmous,11  
SY100300,Safita,12  

With the report counts for each city, we can now get back to shading our map. Let's revisit our approach to rendering the map, and add additional logic to incorporate our new incident data.

d3.json("cities.json", function(err, data) {

  // I know! Nested AJAX calls will be the death of us all. Moving swiftly on, 
  // the important thing to note is that we're retrieving the new syria-incidents 
  // file we created using Node.js and the CrisisNET API. 
  d3.csv("syria-incidents.csv", function(err, incidents) {

    $.each(data.features, function(i, feature) {

      // let's lookup the incident data associated with this feature using the 
      // PCODE we've been storing all over the place. Note that we're using 
      // Underscore.js and its findWhere method. This method will find the 
      // first record in the incidents data that has a cityID equal to the 
      // current feature's PCODE property

      var record = _.findWhere(incidents, { cityID: feature.properties.PCODE });

      // with our record in hand, let's calculate the relative level of 
      // violence in this city by comparing the total number of incidents to 
      // the total number of reports of violence in Syria as a whole.

      var total = parseInt(record.totalIncidents);

      // In real life you would look this up using the API, but for now just 
      // trust me the number is accurate.
      var totalSyriaIncidents = 120000;

      // Get the relative value, then multiply it by 1000 just to make the 
      // number easier to work with.
      var count = Math.ceil((total / totalSyriaIncidents) * 1000);

      // We'll make 12 the maximum level of violence visible on our map
      if(count > 12) { count = 12; }

      // Now that we have a concept of this city's level of violence, we can 
      // assign that count to a css class. This will give each path element
      // a class like count-10
      svg.append("path")
        .datum(feature.geometry)
        .attr("class", "border")
        .attr("d", path)
        .attr("class", "border count-" + count);
    });
});

Finally, now that CSS classes have been assigned to each city's corresponding path element, we can shade those cities to show their relative level of conflict. This is largely up to you, but one approach is to choose darker opacities to indicate higher amounts of violence.

.count-0 {
  fill-opacity: .2;
}
.count-1 {
  fill-opacity: .3;
}
...
.count-11 {
  fill-opacity: .9;
}
.count-12 {
  fill-opacity: 1;
}

Shading your map is only the beginning. Using svg and path lets you create an interative experience using click, mouseover, and all the other events availabe to other DOM elements. For example in the map we created you can click on individual cities to see a recent YouTube video of militants operating in that region.

So now you know, go make some interesting maps!