Don’t stop – there is a lot more to discover out there!

There is a rumour out on the street that says that the age of the Zettabyte is near. Now that is a realistic number when measuring internet traffic. But could there possibly be analytical environments of this size? There are lots of Teradata customers that are in the Terabyte range, and they do unique stuff with their data. Is there any point in having huge data volumes available in itself? A Zettabyte equals 1024 Exabytes, which equals 1024 Petabytes – and the Petabyte club is still an exclusive circle of Teradata customers (although their volumes are increasing at a breath-taking pace.) What is their interest in “big data”? Couldn’t there be a point when large data volumes become an obstacle, rather than an opportunity for analytical purposes?

Listening to the big guys, you don’t get the impression that they worry about size. Large data volumes rather seem to be their playing field on which they try out new tricks. While many data warehouse architects are heavily concerned with economics, these guys are worried that they do not capture enough detail. Oliver Ratzesberger of eBay specifically quoted one of his colleagues who said that you should never throw away any data at all because you never know what you may want them for in a few years’ time. The knack is to have the data ready when the business idea pops up rather than having to wait an extra 12 or 30 months until you have collected enough material to test it on.

This appears to be the attitude that sets the Petabyte club apart from (and often ahead of) other players: they are moving away from using data to confirm suspected answers towards discovering genuinely new things. It requires them to capture high quality detailed data as opposed to summary data. For example, storing information about transactions will enable you to identify some long-term patterns. But it is the interactions with the company that a customer has before and during the purchase that tell the whole story, especially when the sale does not take place in the end (which is clearly the more interesting case.) On websites, simple usability details may turn out to be deal-breakers en masse. But you will never find them unless you track and analyze customer interactions. The club members make sure they can do so when they notice that they need to.

The lesson to learn from these giants is that when you are laying out your data warehouse, it makes sense to look beyond the first few years when you do standard reporting and a bit of analysis. At a later stage, when you have learned to make full use of your insights and when your organization has begun to direct more and more questions at the analytics department, you will want to predict, react to and trigger business events. For example, look at the telecommunications sector. A few years ago, some people questioned the benefits of storing call detail records. The reality today is that if you cannot analyze these data by now, you’ll probably find it hard to stay in business at all, given your competitors’ analytical advantages. The same is happening with OSS data now. This data is turning out to be extremely valuable if you want to understand customer experiences with your services. Are there, for example, any bandwidth bottlenecks in any area that actively drive customers away? And isn’t there any way you can spot these problems in real-time and rectify them right away? Well, there will be, unless you have decided against capturing those data on a detailed level in the first place.

It’s worth to keep this in mind as social media data, and all those other new data sources, are about to be integrated into enterprise analytics. An assortment of data won’t keep you at the head of the field. I am aware that this looks like a massive challenge to many enterprises. If handling the Petabyte was like breaking the sonic wall, handling the Zettabyte appears to be like trying to travel at speed of light. Well, I’d say just don’t stop (as Freddy Mercury has put it), keep on pushing – and we will get there all together! The consensus at the panel was that in the next 3-5 years, we will see the required innovations. Now ain’t that worth keeping the faith?

Socialization at the Partners

Darryl McDonald has elaborated his concept of socialization of data (socializing is a key thing at conferences, so why have we never thought of those data before) today, and if you want to watch him doing so you can see for yourself here. People have expressed their curiosity about this concept in response to my post yesterday, so I will try and depict it here, in the way I understand it.

Darryl basically argued that if you want to draw value from your data, you need to do three things: integrate, explore and share them. Now this is what you do in databases of any size, no matter whether you are running a mere data mart or a full-scale enterprise data warehouse. In my own simple words, Darryl’s message was that we are used to building our own little model worlds – some smaller, some bigger – and tend to be content with them for some time, while at the same time data out there is virtually exploding. The examples he gave were social media, mobile devices and sensors. All of these new data sources hold huge business potentials once enterprises start to tap into them, which is already happening, and will be even more of it in two years’ time or so. What Darryl was really telling us was that we should think up ways to find and grasp those new opportunities. Teradata is providing the technical means for it.

What it all comes down to is what I call the “realization gap”, which occurs because we have been preached in the past years that (textbook style) innovation is led by business: we need it, so we invent it. But I think that at the moment, changes in business will be driven by new technologies, not the other way round. Just remember that mobile devices and geospatial data came first and all those location-based services that we appreciate now followed. In fact, only 18 months ago Teradata CTO Stephen Brobst predicted, not reported, these services. This gap has been closing with frantic speed.

Back to the technical side, the three steps of integrating-exploring-sharing your data. There was a challenge for us that we have risen to. On the one hand, we know that deeper insights require integration of new data with existing data – for example, making customers suitable offers requires more than just their location data, obviously. On the other hand, there is a whole new diversity both of data types and analytical tools specializing on these types. This is why Teradata has put so much effort on integration with partner tools and why we are looking for new ways in which our platforms can work together. From in-database analytics, data labs and the unified LDM to the alignment with Hadoop systems, it all serves to support the vision of an enterprise ecosystem that is open for data and people, as Darryl put it.

So we at Teradata are enabling tomorrow’s business breakthroughs because we have a clear idea which data are being generated and because we are making them available for analysis. Darryl’s platform socialization of data is our bit to help close the realization gap and at the same time take the pulse of all those pioneers that brace themselves to take those new opportunities. After all, ideas get born when people with different backgrounds engage in discussion, don’t they?

So much for now. If you want some lighter inspiration, what views you can get from truly granular material, have a look at the Sand Artists who performed at today’s general session.

Time-travelling with your data warehouse

This headline is not an allusion to the inevitable jet lag that many visitors from our region still feel. It’s about the most spectacular announcement that has been made today at the Teradata Partners conference today, in my view at least. And, as I indicated yesterday, there was a whole wealth of technologically interesting news at today’s media briefing. More on those later.

Think about this: Have you ever wondered why on earth you have taken a decision that turned out to be awfully wrong? If this has ever happened to you, I am sure that you had at least one well-worked out theory. You will say that you didn’t pay enough attention to X or that you were too much preoccupied with Z and, most importantly, you vow to never make this mistake, any mistake again. Data warehousing is all about decision support. So if you want to know why decisions were made in a certain way, they should give a good idea, shouldn’t they? You should be able to simply rewind those data and look at the full view of the time.

This was actually not so easy for various reasons. Data change over time, some of the data simply gets overwritten, and this is why we used to take snapshots of these data every now and again. But this is not enough to capture when exactly certain changes occurred and what effect this had on dependent data. This used to require a lot of “manual” work (complex coding, really) that is now being done by the Teradata Database 13.10. The effect will be much like with geospatial in the recent past: everyone relied on zip codes because geographical representation was such a hustle – and suddenly every mobile application uses it. So it bet that the temporal rewind button will be everywhere in the near future. And you can use it to find out what you really did know when you took a certain decision. You can even use it to demonstrate this to third parties if you want to. And there are lots of business questions that you can answer a great deal more easily and accurately with this feature. I find this whole idea really fascinating, maybe because I remember the movie Back to the Future so well. It was released in Italy exactly 25 years before Teradata’s temporal data announcement. If I could rewind my life to the 80s, I would probably be having a good time listening to an U2 vinyl. I wonder whether I would feel more like “Marty McFly” (Michael J. Fox) who doesn’t really grasp what is happening to him or like “Doc” who feels completely in control all the time.

- Teradata offers a Unified Logical Data Model now that allows enterprises to map all of their value chains from beginning to end (and possibly crossing boundaries of industry that were previously catered for with distinctive LDMs)

- More performance, storage and usability with the platform family, which has a full five members by now

- Teradata Relationship Manager has incorporated modules to engage customers on smartphones, iPads and social media sites within seconds – with all the knowledge stored in the data warehouse at hand

- Teradata Accelerated Analytics Portfolio includes lots of new data types, emerging technologies, partner optimizations, and tools for application development

And there is also some news from our partnerships that I won’t be able to cover today. As I have said, there was a lot. And hey look, I can’t go massive-parallel. I’m a mere human being – and I plan to keep it that way for the foreseeable future. But I pledge that I’ll come back to you as soon as I can. See you.

Another Euphoric State Conference

I have arrived in San Diego at this year’s Teradata Partners Conference now, and as usual when I reach the West Coast of the United States I feel somewhere in between a terrible jet lag (the difference to Milano and most of Europe is nine hours) and feverish excitement that tells me to just shrug off the fatigue and enjoy. When I flew in on Saturday, I found myself humming “it never rains in Southern California” to the San Diego rain and some people even seemed to nod at me in silent agreement. Now, who was it again that referred to California as “Euphoric State?”

The only real calamity that plagues this piece of land (apart from the occasional earthquake) seems to be wildfires that break out from time to time. In fact, back in 2007 a wildfire raged in the area near Rancho Bernardo where both the Teradata Research and Development and the Global Support Center (GSC) reside. The authorities wouldn’t run any risk and ordered an evacuation that forced Teradata to close its facilities temporarily. They were not harmed, and a few days after the fire (and the smoke) was gone, operations resumed as usual. In the meantime, the GSC was facing a major challenge, having to ensure that Teradata’s 24/7 customer support would be kept up. It turned out that its business continuity plan worked splendidly and operations progressed despite of the fires. For example, when a retailer reported performance challenges at one of its nodes, Teradata Customer Services were on cue and solved the problem as speedily as usual. The GSC was rising to the challenge alright.

The proximity to Teradata Research and Development at this year’s conference makes sense because – I think I can reveal that much – that there will be a lot of technologically relevant announcements from Teradata and its partners on Monday. During the next few days, I plan to follow around some journalists the Expo area and report back to you which partners they are most interested in and what questions they debate. The same goes for the various conference highlights, most importantly the general session on Monday where our Chief Marketing Officer Darryl McDonald will explain his concept of the socialization of data.

So much for now. Let the show begin!