Making The Insights For Log Data Simply Accessible

Trevor Parsons

Subscribe to Trevor Parsons: eMailAlertsEmail Alerts
Get Trevor Parsons: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Related Topics: Big Data on Ulitzer

Blog Post

How to Avoid the #BigData Black Hole | @BigDataExpo [#IoT]

It's hard to avoid the world of opportunities that has been opened in data collection and analysis

It takes a lot of will power, in our data obsessed world to say "too much!" However, there are many ways where too much information is destroying productivity, and actually causing bad decision making, not good. But it is hard to avoid the world of opportunities that has been opened in data collection and analysis. So how do you balance the two? The first step is to understand there is a big difference between data collection, and it's utilization. While it seems subtle, the difference is key, and utilization is where many make mistakes.

Many confuse data collection and data utilization as the same thing, or at least being very similar.
Collection merely indicates the storage and analysis of data by automated systems, maybe never viewed by human eye or mind. It is when you spend the time to access and understand the data collected where there is a human cost.

"When information is cheap, attention becomes expensive." - James Gleick

New tools mask human effort by taking raw data (in hard to consume formats) and making it much easier to understand, and more importantly; query upon. This makes users feel like they need to look at it all, and the UI is saving them all the time they need. But, in most cases, we did not previously collect this information, nor analyze it. So the opportunity is higher, but any effort is net new.

big-data-blackhole

When I get in front of a great log management tool, I sometimes get that same feeling as when I get addicted to a new iPad game; crushing candies, or tending to farms. Because there is a sort of gamification when you dig, and if I just dig a little deeper I will reach the goal. Next thing you know, you have spent three hours without any results.

"Information is not knowledge and knowledge is not wisdom." - James Gleick,

It is a people problem too
There is effort required to first understand, and then act on the information we receive. But the enthusiasm for learning more, and "information is power" creates a sort of blindness. A blindness that skirts the need to quantify our effort spent in analyzing the data against the results (decisions, actions items) that from it. To make things worse, the amount of data it takes for one decision is often way more than we realize.

So, what is the impact of spending too much time trying to utilize the new pile of information?

  • Time suck: Being taken away from daily tasks, by asking a question of the platform that takes way more time than expected.
  • Dwelling on details that do not impact the business: Leading to zero actionable results.
  • The anchoring effect: If you are familiar with the psychological anchoring effect, where new decisions are based on an existing foundation of positions, and opinions. With all the data in your log analysis platform, if you are not aware of anchoring, you will the ask questions that produce the answers you seek.
  • "That is what the numbers say ...." Platforms that only produce time-series dashboards and numbers lead to the habit of using this data as an default answer to all questions, and deflecting mechanism. It's easy to put the decisions on the system, instead of the people who were previously responsible for them.

These are not light topics I know, but I have seen time and time again where the abuse of data, and the time wasted on collecting it eats organizations from within.

Does the tool save your time or steal your eyeballs?
Part of this whole situation has a lot to do with the tools. Tools can be good or bad at surfacing information. All the latest APM, Log Analysis, Error Logging, and usage analytics tools have ways for you to dig-in deep. Putting the burden of when enough is enough on the user. Who often is too absorbed to keep track.

In fact, many of these tools encourage you to go deep, eventually spending hours sifting thought information that provides no real value real, but perhaps a lot of little "oh that is interesting". Those little hits are just like pulling the lever on a casino machine. If you have to become specialized to use the platform, and make it a large portion of your daily effort, you are committed. After all, churn is a major concern for tech companies. But should it be?

"Business is not just doing deals; business is having great products, doing great engineering, and providing tremendous service to customers." - Ross Perot

A great tool should help you get the data, make decisions (or prepare them for the expanded team), and move on.

It should prevent you from entering the data black hole, and guide you to what is most useful. The tool should also put a layer between you and the data, not to encourage the diving-in too deep which we know can happen quickly.

Here are some ways to avoid these problems:

  • Hire a dedicated analysis person: Definitely a nice-to-have. But really, this is just a workaround for a bigger problem.
  • Ask discrete questions: If you keep your questions simple and small, then you can move faster. But we all know that there are bigger questions we want answered.
  • Plan your logs in advance for utilization, not collection: If you think about how you want to use the information, instead of just gathering data, you will make the analysis go much faster. In a crunch, this seems like a wasted effort.
  • Focusing: There are a lot of "Fun" questions you can ask. But time is better spent on questions decided-on in advance and related to current initiatives and activity. At the same time, wouldn't it be nice for the data to tell you what is interesting without you having to predict it in advance?
  • FIND THE RIGHT TOOL! Many analysis tools only help "send information and put in a nice format ", aka data collection. While this is a must, the focus of the tool should be on utilization. And this should be their roadmap focus.

Find it, act on it, move on
Part of creating this value layer is discovering what is happening, but also what is not happening, and what is out of normal. Watching the flow of data can be memorizing, but decisions are made on things that are not consistent with the steady state of systems and applications. Or with sudden changes. Things that normally occur in the system, but for some recently did not.

Data collection should be synthesized into meaningful events. Getting users addicted to a platform by the quality and frequency of decisions versus encouraging them to spin the wheel to see what happens and becoming a 5th limb.

First, admit to yourself that there is too much information, more than you could every analyze on your own. And then find a tool that takes over analysis, and only shows you what is valuable so that you and your team can get on with your day.

Conference Schedule Announced

Are you ready to put your data in the cloud?

What is the future of security in the cloud?

Does Docker quickly advance the development of an IoT application?

What are the implications of Moore's Law on Hadoop deployments?

Get all these questions and hundreds more like them answered at the 15th Cloud Expo, November 4-6, 2014, at the Santa Clara Convention Center, in Santa Clara, CA. The Cloud Expo / Big Data Expo / @ThingsExpo / DevOps Summit programs are now available for you to inspect and investigate in advance.

Our upcoming November 4-6 event in Santa Clara, California will present a total of 10 simultaneous tracks by an all-star faculty, over three days, plus a two-day "Cloud Computing Bootcamp" presented by Janakiram MSV, an Analyst with the Gigaom Research analyst network, where he covers the Cloud Services landscape.

Cloud and Big Data topics and tracks include: Enterprise Cloud Adoption, APM & Cloud Computing | Hot Topics, Cloud APIs & Business, Cloud Security | Mobility, Big Data | Analytics.

@ThingsExpo content tripled from a single track in New York to three simultaneous tracks: Consumer IoT, Enterprise IoT, IoT Developer | WebRTC Convergence.

DevOps Summit also doubled from a single track in New York to two simultaneous tracks: "Dev" Developer Focus and "Ops" Operations Focus.

Schedule for Cloud Expo / Big Data Expo / @ThingsExpoHere

Schedule for DevOps SummitHere

Now that we have published the full conference schedule, please check back for daily updates as we finalize new session abstracts by working with our distinguished faculty members. For your questions please contact us at events (at) sys-con.com. Last but not least we will announce our keynotes on the hottest subjects to be delivered by world-class speakers!

The largest 'Internet of Things' event in the world has announced "sponsorship opportunities" and "call for papers."

The 1st International Internet of @ThingsExpo was launched this June at the Javits Center in New York City with over 6,000 delegates in attendance. The 2nd International Internet of @ThingsExpo will take place November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, California, with an estimated 7,000 plus delegates attending over three days.

@ThingsExpo is co-located with 15th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading IoT industry players in the world. In 2014, more than 200 companies will be present on the @ThingsExpo show floor, including global players and the hottest new technology pioneers.

Sponsorship and Exhibit Opportunities for @ThingsExpo Silicon Valley and New York Are Now Available
Sponsors and exhibitors of Internet of @ThingsExpo will benefit from unmatched branding, profile building and lead generation opportunities through:

  • Featured on-site presentation and ongoing on-demand webcast exposure to a captive audience of industry decision-makers.
  • Showcase exhibition during our new extended dedicated expo hours
  • Breakout Session Priority scheduling for sponsors that have been guaranteed a 35-minute technical session
  • Online advertising in SYS-CON's i-Technology publications
  • Capitalize on our comprehensive marketing efforts leading up to the show with print mailings, e-newsletters and extensive online media coverage.
  • Unprecedented PR Coverage: Editorial coverage on IoT.sys-con.com, Tweets to our 75,000 plus followers, press releases sent on major wire services to over 500 combined analysts and press members who attended Internet of @ThingsExpo - making it the best-covered "Internet of Things" conference in the world

For more information on sponsorship, exhibit, and keynote opportunities contact Carmen Gonzalez by email at events (at) sys-con.com, or by phone 201 802-3021. Book both events for additional savings!

@ThingsExpo Silicon Valley (November 4-6, 2014, Santa Clara, CA)
@ThingsExpo New York (June 9-11, 2015, New York, NY)

Secure Your VIP Pass to Attend @ThingsExpo Silicon Valley
Internet of @ThingsExpo announced today a limited time free "Expo Plus" registration option. The onsite registration price of $600 will be set to 'free' for delegates who register during September.

To take advantage of this opportunity, attendees can use the coupon code "IoTSeptember" and secure their "@ThingsExpo Plus" registration to attend all keynotes and general sessions, as well as a limited number of technical sessions each day of the show, in addition to full access to the expo floor and the @ThingsExpo hackathon.

The registration page is located at the @ThingsExpo site here.


@ThingsExpo New York 2015 'Call for Papers' Now Open
The 3rd International Internet of @ThingsExpo, to be held June 9-11, 2015, at the Javits Center in New York City, New York announces that its 'Call for Papers' is now open. The event will feature a world class, all-star faculty with the hottest IoT topics covered in three distinct tracks.

Track 1 - Consumer IoT and Wearables: Smart Appliances, Wearables, Smart Cars, Smartphones 2.0, Smart Travel, Personal Fitness, Health Care, Personalized Marketing, Customized Shopping, Personal Finance, The Digital Divide, Mobile Cash & Markets, Games & the IoT, The Future of Education, Virtual Reality

Track 2 - Enterprise IoT: The Business Case for IoT, Smart Grids, Smart Cities, Smart Transportation, The Smart Home, M2M, Authentication/Security, Wiring the IoT, The Internet of Everything, Digital Transformation of Enterprise IT, Agriculture, Transportation, Manufacturing, Local & State Government, Federal Government

Track 3 - Developer IoT: WebRTC, Eclipse Foundation, Cloud Foundry, Docker & Linux Containers, Node-Red, Open Source Hardware, Leveraging SOA, Multi-Cloud IoT, Evolving Standards, WebSockets, Security & Privacy Protocols, GPS & Proximity Services, Bluetooth/RFID/etc., XMPP, Nest Labs


@ThingsExpo billboard is viewed by more than 1.3 million motorists per week on Highway 101, in the heart of Silicon Valley

Help plant your flag in the fast-expanding business opportunity that is the Internet of Things: Submit your speaking proposal today here!

Download @ThingsExpo Newsletter Today Here

Chris Matthieu Named @ThingsExpo Tech Chair

Internet of @ThingsExpo named Chris Matthieu tech chair of Internet of @ThingsExpo 2014 Silicon Valley.

Chris Matthieu has two decades of telecom and web experience. He launched his Teleku cloud communications-as-a-service platform at eComm in 2010, which was acquired by Voxeo. Next he built an open source Node.JS PaaS called Nodester, which was acquired by AppFog. His latest startups include Twelephone. Leveraging HTML5 and WebRTC, Twelephone's BHAG (Big Hairy Audacious Goal) is to become the next generation telecom company running in the Web browser. Chris is currently co-founder and CTO of Octoblu.

Website: http://www.ThingsExpo.com

Twitter: http://www.Twitter.com/ThingsExpo

About SYS-CON Media & Events
SYS-CON Media (www.sys-con.com) has since 1994 been connecting technology companies and customers through a comprehensive content stream - featuring over forty focused subject areas, from Cloud Computing to Web Security - interwoven with market-leading full-scale conferences produced by SYS-CON Events. The company's internationally recognized brands include among others Cloud Expo® (CloudComputingExpo.com / @CloudExpo), Big Data Expo (BigDataExpo.net / @BigDataExpo), DevOps Summit (DevOpsSummit.sys-con.com / @DevOpsSummit), Internet of @ThingsExpo (ThingsExpo.com / @ThingsExpo) and Cloud Computing Bootcamp (CloudComputingBootcamp.com).

Cloud Expo® and Big Data Expo® are registered trademarks of Cloud Expo, Inc., a SYS-CON Events company.

More Stories By Trevor Parsons

Trevor Parsons is Chief Scientist and Co-founder of Logentries. Trevor has over 10 years experience in enterprise software and, in particular, has specialized in developing enterprise monitoring and performance tools for distributed systems. He is also a research fellow at the Performance Engineering Lab Research Group and was formerly a Scientist at the IBM Center for Advanced Studies. Trevor holds a PhD from University College Dublin, Ireland.