The growth in data over the past few years has almost outstripped our ability to handle it effectively, affecting storage, management, compliance and decision-making.

Corporate information management now demands that companies have processes and techniques for dealing with rapidly increasing volumes of business-relevant data. Historically, data and information have been seen as a cost to the business, but that is no longer the case.

Ian Charlesworth, principal analyst at Ovum, says that information now ranks second behind people as an organisation’s greatest asset. “With that conclusion must come a level of corporate investment and responsibility for its management, retention and safekeeping,” he says.

Richard Dawson, IT services manager at Bracknell Forest Borough Council, has experienced a 60 per cent increase in data handling over the past five years. “Electronic social care records, together with electronic planning applications and the abundance of digital media, has caused a significant increase in the amount of data we now hold,” he says.

“The data explosion is also manifested in our servers, which have increased from less than 100 three years ago to more than 300 today, to cope with the extra demand for information and processing. While the number of databases has also increased, these are now being consolidated into fewer, but larger, systems.”

Such trends are not an uncommon problem, as confirmed by Clive Jones, director of customer services at Mid Beds District Council. He says the demand for data storage has rocketed, though luckily the price per megabyte has dropped.
“At Mid Beds, we have gone from half a terabyte to three terabytes of storage and from 10 servers to more than 40,” he says.

Duplication of information is a big problem in corporate systems, especially within email applications, and one caused primarily by the ease of adding attachments which can significantly increase the data load, even on a small circulation list.

Steve Ireland, senior records and information manager for the Northern Ireland Prison Service, says that everyone who gets a copy of an attachment tends to keep a copy, and that this needs to be managed better if the data burden of today’s email systems is to be reduced.

Medical recruitment company HCL, for example, has candidate CVs stored in at least three or four different places across its network, and group IT manager Luke Harding says duplication is a bit of a headache. “We tend to throw more disks at the problem,” he says.

“Currently we have one terabyte per fileserver and I estimate that will grow by 50 per cent over the next two years. There is no quick way to combat that growth. Even archiving is not the answer for the group ­ my users want everything
available now, even if they haven’t used it in the past six months.”

For Darlington-based gas and oil engineering company Whessoe, the use of technology has also brought information benefits and challenges, according to business systems manager Wilf McNaughton.

“We depend on detailed drawings, complex calculations and models to design and construct gas and oil terminal facilities around the world. Some 15 years ago our computers didn’t have the power and storage capacity,” he says. “Today iterative finite element analysis calculations can generate hundreds of 50MB files, quickly creating vast amounts of data.”

For Whessoe, collaboration is a big factor in its information management process. A recent gas installation project was designed in Darlington, Houston, San Jose, and Oslo before being fabricated in Korea, sub-assembled in Spain and finally installed in Italy. “And that takes some teamwork ­ not to mention good communication and information management processes,” says McNaughton.

Computerised 3D-modelling has simplified visualisation and clash-checking, partly replacing 2D computer aided design (CAD) drawings.

“A CAD drawing might be 0.5MB. A 3D model could now be 4GB, holding dimensional and materials data for every component of a multimillion-dollar installation,” says McNaughton.

“And then, of course, there are all the revisions throughout the design and construction stages ­ it all adds up to the creation of a vast amount of data that has to be managed.”

But for users there is no universal approach to information management, and the problem is greater than a single vendor would wish to support. And while mergers between vendors are starting to blur the boundaries and technologies are maturing ­ especially regarding management of unstructured data ­ the challenge of information management remains.

To handle the growth of data, organisations have adopted diverse technologies, including the use of storage area networks (Sans) and electronic document record management systems (EDRMS), and implemented strategies ranging from knowledge management to information lifecycle management (ILM). But all these efforts remain in independent isolation.

“Physically, we have a San and have rolled out EDRMS, with retention periods on folders and appropriate security access,” says Mid Beds Council’s Jones. “We have an information management policy and guides for content creation for the various channels.”

However, for most organisations the issues arise from the growth in unstructured or semi-structured data. Ireland, for example, acknowledges the organisation has failings in the way it manages information, and is taking exhaustive steps to rectify the problem.

“Compliance is partly driving the trend and we had a wake-up call with the Freedom of Information Act,” he says. “A lot of business areas have unstructured data that is difficult to pull together.”

With 300,000 paper files plus all the electronic data to audit and categorise, the prison service is undertaking a huge, but necessary, task.

The move from paper files to electronic storage is a common trend for many companies, and can add to the burden of finding and retrieving information.

International law firm Allen and Overy has devised its own global virtual file system, to capture and retain the unstructured information found in emails, word processed documents, faxes and scanned surface mail, as a way of controlling its own risks in the face of burgeoning volumes of data and the need to keep its critical information reliably retained.

Heather Webster, head of global IT support, says with a global virtual file, the organisation has one area to tackle.

“We are in the process of implementing a global archiving system that will take closed virtual files and move all content to a near-line or tier-3 online storage,” she says.

In the next few years there will be a shift in the way that information is managed across organisations. Previously, boundaries were placed on data because of restrictions imposed by the technology.

“The different approaches on the management of structured and unstructured data are being chipped away,” says Ovum’s Charlesworth. “So we can start to treat all our data in a similar way.”

Acquisitions among the major vendors will also start to offer more interoperability in the way that data is managed, while emerging standards around web services, Java Database Connectivity (JDC) and XQuery will facilitate interoperability.

For users, the big issues that need to be solved over the next years include enterprise search facilities, archiving technologies and full application integration and collaboration.

True information sharing between systems internally and, more importantly, between partner organisations must happen, especially as more data will be presented publicly and in alternative formats such as video and audio streaming.

Information management is a permanently moving target. However, according to Ovum, help may be at hand ­ and the advent of unified information management (UIM) will bring a much more business-focused message about approaches and techniques.

UIM will help IT managers deal with rapidly increasing volumes of business-related data that may be stored in any number of formats or locations as well as being readily accessible to support business processes and decision-making.

Do not give up. The continuing growth of data might be manageable after all.

Watch our video on information management at: www.computing.co.uk/tv