Data mining the data explosion

Organisations are able to create new opportunities to capture information from wide varieties of data and content which is coming at them in huge volumes and at ever increasing speed. This information includes the logs of application and web site usage, social networking sites, collaborative tools, documents and multi-media and more, augmenting the transactional records which support an organisation’s traditional operations.

There is a shift of power to the consumer and citizen which is forcing organisations to understand and anticipate their behaviours and needs based on individual insights across all channels. Capability to derive insight in real time will be needed where multiple data sources are joined together and correlated so that responses can be executed in a timely manner.

But there are challenges. Individuals enthusiastically share personal details of the current activity and whereabouts on social networking sites to facilitate their personal and social lives. However open data can be used in data mining for commercial exploitation through targeted marketing. Many individuals are uneasy when they see organisations deriving deep personal insights about them to tailor offers, promotions and services, especially when sensitivities are explicitly revealed. Mining of non-anonymised data can undermine privacy.

There is uncertainty over data ownership. If individuals provide data about themselves to companies, or enter it into systems to use services which they provide, who owns that data? Who owns information captured on individual use of a smartphone or internet search history? The UK intends installing smart meters in households: does the consumer or the energy company own information on energy usage? For each of these areas how is data secured and access governed?
If an individual wants a trace of part of their life, for example to prove a claim or aid a defence, is it possible to retrieve the relevant information for organisations? Such action might require phone records, web usage, travel and location information, and purchasing history.

Many smartphones and social media sites are supported by a platform provided by a commercial organisation in which individuals store details of their contacts, their photos, video clips, music and more. How feasible is it to extract all this information and migrate it to an alternative provider if you choose to switch smartphone platform provider?

Large quantities of data also reside within government and HMG has a desire to make non-personal data available for others to use – both freely and as a purchasable services – for wider public benefit. Commercial organisations can use this data to create new and enhance existing services which they in turn offer to their customers. Success will necessitate an understanding of the context and currency of data which government is publishing. A meta model will be needed to help consumers understand how they should use the data provided. Data quality and trust a key considerations, and further thought will need to be given to the liability for use of erroneous data and mis-use.

Many areas of uncertainty and unresolved legal and ethical considerations for individuals, commercial organisations and governments remain to be explored and addressed.