Add Thematic Analysis- Cruncher

[ekastimo]

Background

In an effort to improve access to financial services to the population in Uganda, HOT has been participating in a project attempting to map a different kind of financial service providers in Uganda. Financial service providers are entities that provide services like deposits, credit, transfers, withdrawals, foreign exchange etc, both in the formal and the informal sectors. In Uganda, examples are banks, ATMs, microfinance institutions, mobile money agents and Savings and credit co-operatives (SACCOs). Mapping them and viewing them through different thematic maps will allow different stakeholders such as the financial service providers themselves, the general public, regulators, donors and others to identify what areas are underserved or where there is potential for rolling out new coverage. The thematic analysis is built into the OSMA framework, and as such also provides an example of how thematic analysis can be added to OSMA.

Overview

HOT chose four thematic views which each answer a specific question that financial service providers might ask. The main goal was to create an analysis tool that can provide a basis for decision-making, mainly for financial service providers to see where there could be a potential market (population density and economic activity), where their competitors are active or where there are areas that do not have existing FSPs. The same approach can be chosen for other thematic analysis; however, it is crucial that the information showed on the maps is already providing a certain degree of analysis when it loads. The analysis can then be refined or adjusted using the controls and parameters available. It was somehow challenging to find a good balance between providing sufficient analysis for the maps to be good decision-making tools, but at the same time not be too complex for the users to understand.

Thematic analysis and context data

There is great potential for thematic analysis using the data from OSM. Our technical proposal outlines how we made improvements in OSMA to allow for easier thematic analysis. In addition, good knowledge of the OSM keys is necessary in order to know what data could be relevant for the given user group. We also chose to introduce some external datasets to provide context, such as Worldpop.

However, there is a trade-off between using external datasets and data available in OSM. Population data can be re-downloaded from Worldpop and updated in case there is newer data.

For economic analysis, we chose to use a composite of data available in OSM in order to avoid the need for updating. After some trial and error, we used the following keys to show economic analysis

• Number of buildings
• Distance from the main road
• Number of people (This came from Worldpop)

This is what gave the best result in Uganda, however, note that it might not give the same result in other areas due to the different nature of data available there. Other than that, we encountered the following challenges with the data for financial service providers available in OSM:

• Clear conventions were not followed during data collection: example:
  The operator property on some of the banks was empty while in other cases, some bank names were saved in short form, i.e. DTB was used in many cases instead of Diamond Trust Bank and this means that not all Diamond Trust Bank branches are rendered when you select it since some are saved under DTB.
• Mobile money agents are do not necessarily belong to an operator and the majority have services of up to 3 telecom companies. This made it quite hard to do comparison on mobile operators
• Data for financial service providers in Uganda is predominantly available in Eastern Uganda and Kampala because this is where the data collection exercises have taken place. For now, this, therefore, gives a distorted picture of the reality on the ground. We are hoping that as the data collection efforts continue, this will improve.

Proposed by

Timothy Kasasa (Laboremus), Benjamin Lutaaya (Laboremus)

The Problem

While there are over 5000 Tags in OpenStreetMap, currently, there are only 3 tags configured for analysis in OSMA. There could be several reasons but the major cause is the fact that the process of adding a new Tag is quite complex and fragile. One would have to go through 5 step and edit 4 files just to have a new tag. An example is adding waterways here

• Create a filter configuration for the Tag
• Add Code to the run.sh script to enable crunching
• Edit oqt-user-experience/map.js file to add static experience initial value
• Edit oqt-user-experience/index.js file to add static experience initial value

Furthermore, in order for the cruncher to support thematic analysis, there is a need to make the code highly dynamic and configurable such that it allows for easier and more scalable addition of features to the backend, reducing the amount of code and configuration needed to change exposed features. these datasets should also be easy to integrate with an API.

The solution

This proposal is heavily supported by the front-end proposal and all the changes and context are highly dependant on the front end implementation.

OSM-Cruncher architecture

---

A proper understanding of the generic architectural flow of the OSM cruncher is vital in appreciating the refactoring required to implement thematic analysis within the existing system. This process can be broken down into three major steps.

Filtering phase

At this point the entire world dataset from OSM if filtered to obtain the required TAG. The output is a minimal .mbtiles file

Binning and aggregation

AT this stage, OSM features are grouped into geocells for analysis.

Downscaling

At this stage, the geocells are aggregated to generate tiles at a lower zoom level, for example, zoom level 12 mbtiles are aggregated to obtain zoom level 11 tiles

A more detailed cruncher architecture can be found here

Code Base

Adding a new Tag

As described above, this process is very cumbersome, however, with the new implementation; we made a couple of additions and refactorings to greatly improve this process. A user only needs to edit one place and the rest of the process is automated, i.e. Compared to the fomer 4 steps, current to add a new tag for example power;

• Create the config file, and place it in fsp-config folder with the format below:

{
    "name":"power",
    "geometry": "Point",
    "tag": "power",
    "factor": 32,
    "experience": {
        "file": "./experiences.json",
        "field": "power"
    }
}

And that's all, during the crunching process, the cruncher automatically picks up this configuration file and creates a .mbtiles file in the results folder. This will also be automatically be picked up by the tile server and made available for the client to consume.

Thematic analysis

WIth the new dynamic configuration, we are able to create even more complex filters and consequently do very detailed analysis on OSM data. For example, we wanted to be able to find out the economic activity of given areas in Uganda. For this, we needed a couple of datasets.

• Number of buildings
• Number of roads
• Number of mobile money agents
• Population density

To this end, we created two filter files.
A json configuration that is passed to the filter phase (as described above) of the cruncher

{
    "geometry": "Point",
    "id": "popnbankatm",
    "composite": true,
    "tags": [
        "building",
        "highway"
    ],
    "amenities": [
        "mobile_money_agent"
    ],
    "fsp": "qn3"
}

Filter script, which is passed to the aggregation and downscale phases

const config = {
    aggregate: {
        sum: [
            { name: '_buildingCount', prop: '_buildingCount' },
            { name: '_noOfMMAgents', prop: '_mobile_money_agentCount' },
            { name: '_highwayCount', prop: '_highwayCount' },
        ],
        population: true,
        economic: true,
        divisors: [
            { name: '_peoplePerAgent', divident: '_noOfMMAgents', divisor: '_population' }
        ]
    },
    downscale: {
        max: ['_noOfMMAgents', '_population','_economicActivity','_peoplePerAgent'],
    }
}

During the crunching process, these file is picked dynamically and processed. the output tile data is then consumed by the client side OSMA to render this map. As a financial service provider, I can then decide to or not set up a mobile money agent within that radius. This decision will, of course, be based on how many people are found in that radius as well as the type of economic activity. This map visualizes how mobile money agents are distributed relative to population and economic activity. The map initially renders grid cells with people per mobile money agents.

Serving tile data to the front end

All the generated tiles and geojson files are stored in the results folder. We modified the server script to dynamically picked up any .mbtile file in this folder and make it available for the client as pbf with the URL format {{server}}/tag/{z}/{x}/{y}.pbf

The server also statically exposes JSON Files int the results/json directory which can be used for point analysis.

API

To demonstrate the ease with which this data can be used with an API, we developed a simple API, to compute the number of mobile money agent in a given distance from a bank or atm

{{server}}/distance/${from}/${to} 
where from = Minimum distance, to= Maximum distance

This simple API is used by Map2 which visualises the overall distance from mobile money agents to a specific (operator) bank. Both the presence within [distance] is important, but also underserved mobile money agents; who are outside of [distance] of any/specific operator bank.

Updating data

All the data generated can be updated whenever HOT has mapped out more features to give a more current analysis to the user of this tool. See x-run.sh for an example invocation of the scripts above and integration with the example server.

Finacial service providers

The list below shows the list of OSM amenities that we chose for our analysis

FSP	OSM type	Value
Mobile money agent	amenity	mobile_money_agent
Banks	amenity	bank, banking_agent
ATM	amenity	atm
MDI	amenity	microfinance_bank
Credit Institution	amenity	credit_institution
MFI	amenity	microfinance
SACCO	amenity	sacco
Bureau de Change	amenity	bureau_de_change
Money Transfer Services	amenity	money_transfer
Post Office	amenity	post_office

Challenges and open issues

• The major issue that is still open is Zoom levels and feature aggregation
• Data for financial service providers in Uganda is predominantly available in Eastern Uganda and Kampala because this is where the data collection exercises have taken place. For now, this, therefore, gives a distorted picture of the reality on the ground. We are hoping that as the data collection efforts continue, this will improve.

References

• OSM Tag Info https://taginfo.openstreetmap.org/about
• World Pop http://www.worldpopdata.org/