[hemmerling] Logfile Data Processing / Logfile Analysis / IT Operations Management

Related page:

• Mathematical Engineering.

• Online Hackathon Devpost "Splunk Apptitude App Challenge", 2015.

• The OpenSource Elasticsearch BV, "Logstash" in Ruby - “Collect, Enrich, and Transport Data”.
  • Elasticsearch BV, - Downloads "Logstash".
  • Elasticsearch BV "Community".
  • Blog Elastic Blog.
• GitHub "elastic/logstash" - “logstash - transport and process your logs, events, or other data”.
• Twitter "elastic, @elastic".

• Rocana.
  • Here you may order te free PDF e-book Rocana "e-Book: Using R.E.A.L. Big Data for IT Operations".
    • The free PDF e-book Using R.E.A.L. Big Data for IT Operations ( PDF ).
• Experts consider Rocana as more advanced alternative to Splunk.

• The Java application Splunk - “Operational Intelligence, Log Management ...”.
  • Splunk "Log-Management-Lösungen: Auswertung von Logdaten für Einblicke in die Vorgänge im Unternehmen" - “Splunk bietet die branchenführende Software für die Konsolidierung und Indizierung von Log- und Maschinendaten, einschließlich strukturierter, unstrukturierter und komplexer, mehrzeiliger Anwendungslogdaten”.
• The free Splunk "Splunk Universal Forwarder" for Windows, Linux, MacOSX,...
• GitHub "Splunk" - As Splunk is not OpenSource, there are just a few additional tools published here...
• Splunk Community.
  • Splunk Developer Portal.
  • Splunkbase - Splunk's App store.
• Wiki Splunk Wiki.
• Blogs Splunk Blogs.
  • Blog Splunk Blog - Category "Dev".
• Twitter "Splunk Dev, @splunkdev".
• Download and do not uncompress the tutorial data file!

• The Splunk Web interface is at http://localhost:8000.
  • username: “admin”.
  • password: “changeme”.

• Book Vincent Bumgarner "Implementing Splunk: Big Data Reporting and Development for Operational Intelligence", 2013.
• Book David Carasso "Exploring Splunk", 2012.
• Book Josh Diakun, Paul R Johnson, Derek Mock "Splunk Operational Intelligence Cookbook", 2014.
• Book Grigori Melnik, Dominic Betts "Building Splunk Solutions (Second edition): Splunk Developer Guide", 2015.
  • Accompanying website Splunk Developer Guidance.
• Book James Miller "Mastering Splunk", 2014.
• “Splunk Developer's Guide.
  • Book Betsy Page Sigman "Splunk Essentials", 2015.
  • Book Erickson Delgado, Betsy Page Sigman "Splunk Essentials - Second Edition Kindle Edition", 2016.
• Kindle E-Book Kyle Smith "Splunk Developer's Guide - Second Edition", 2016 ( no paper edition yet available or announced ).
• Book Peter Zadrozny, Raghu Kodali "Big Data Analytics Using Splunk: Deriving Operational Intelligence from Social Media, Machine Data, Existing Data Warehouses, and Other Real-Time Streaming Sources", 2013.

• Splunk "Let's get started".
• Splunk Documentation, Manuals "Splunk Light" - “Splunk Light delivers full-featured log search and analysis for small businesses and workgroups”.
• Documentation as online HTML website & PDF.
  • Splunk Knowledgebase "Search Tutorial".
    • Splunk Knowledgebase "Search Tutorial" / "Identify event patterns with the Patterns tab".
    • Splunk Knowledgebase "Search Tutorial" / "Search command primer".
  • Splunk Knowledgebase "Getting Data In".
    • Splunk Knowledgebase "Getting Data In" / "Why source types matter".
    • Splunk Knowledgebase "Getting Data In" / "List of pretrained source types".
    • Splunk Knowledgebase "Getting Data In" / "Create source types".
    • Splunk Knowledgebase "Getting Data In" / "Manage source types".
    • Splunk Knowledgebase "Getting Data In" / "Configure timestamp recognition".
    • Splunk Knowledgebase "Getting Data In" / "How timestamp assignment works".
    • Splunk Knowledgebase "Getting Data In" / "Configure timestamp assignment for events with multiple timestamps".
    • Splunk Knowledgebase "Getting Data In" / "Extract data from files with headers".
  • Splunk Knowledgebase "Forwarding Data" -“Install the universal forwarder software”.
    • Splunk Knowledgebase "Forwarding Data" / "The universal forwarder".
  • Splunk Knowledgebase "Knowledge Manager Manual".
    • Splunk Knowledgebase "Knowledge Manager Manual" / "About fields".
    • Splunk Knowledgebase "Managing Indexers and Clusters of Indexers".
    • Splunk Knowledgebase "About Splunk Enterprise regular expressions".
  • Splunk Knowledgebase "Search Reference".
    • Splunk Knowledgebase "Search Reference" / "Evaluation functions".
    • Splunk Knowledgebase "Search Reference" / "rex".
      • Splunk Wiki "Community:RegexTestingTools" - “Some helpful tools for writing regular expressions”.
  • Splunk Knowledgebase "Admin Manual".
    • Splunk Knowledgebase "Admin Manual" / "props.conf".
• Splunk Answers.
  • Splunk Answers - Search for "max_days_ago".
  • Splunk Answers "How does Splunk determine the date, when there is no date stamp in the event?".
  • Splunk Answers "Exclude events with specific field value from results".
  • Splunk Answers "DateParserVerbose - timestamp match is outside of the acceptable time window".
  • Splunk Answers "How to exclude some result".
  • Splunk Answers "Ignoring a specific portion of the log file (header/footer)".
  • Splunk Answers "What's the earliest date I can have in Splunk?".
    • Error message

      "The TIME_FORMAT specified is matching timestamps (Sun Dec 31 00:00:00 2006) outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE.
      Failed to parse timestamps. Default to timestamps of previous event(Wed Dec 31 00:00:00 2014).

    • Answer: 1971-01-01 ( and not 1970-01-01 as you may expect reading EN.Wikipedia "Unix time", or 1970-12-31 ! ).
  • Splunk Answers "How to delete data / index (reset start from scratch)".

• Configuration by:
  • File “C:\Program Files\Splunk\etc\system\local\props.conf”
  • At “Add Data / Set Sourcetype”.

• The ISO date format is easily recognized by the pattern

  Section "Timestamp"
  Field: Timestamp format 
  Value: %Y-%m-%d
  

• Question: “The largest setting I can make for MAX_DAYS_AGO according to the props.conf.spec is 10951 days. Is there anything I can do if I have data prior to 1984?”.
• Answer #1: “You can just set it to -1”.
• Example configurations of “props.conf” & “Add Data / Set Sourcetype / Timestamp” settings:

  Timestamp format = %Y-%m-%d
  Timestamp fields = Datum, Uhrzeit
  

• Example configurations of “props.conf”, “Add Data / Set Sourcetype / Delimiter settings:

  Field preamble = ^#.*
  Field names = Auto
  Field names on line number = (empty)
  

• Example configurations of “props.conf”, “Add Data / Set Sourcetype / Delimiter settings:

  Field names on line number = 10
  Field names on line number = 2
  

• Example configurations of “props.conf”, “Add Data / Set Sourcetype / Advanced settings”:

  MAX_DAYS_AGO = -1
  

sourcetype=vornamen Anzahl_Knaben!=Anzahl
sourcetype="vornamen" Datum="2006-12-31" | top 4 Vornamenstatistik Haeufigkeiten
sourcetype="vornamen" Datum="2007-12-31" | top 4 Vornamenstatistik Haeufigkeiten
sourcetype = "vornamen" Datum != "#" | top 4 Vornamenstatistik Haeufigkeiten by Datum
sourcetype = "vornamen" Datum != "#" | top 4 Haeufigkeiten by Vornamenstatistik, Datum

• Limit of the free Splunk edition: “The maximum file upload size is 500 Mb”.
• Splunk expects CSV data with ”,” delimiter. By default, OpenOffice & LibreOffice save with space ” ” as delimiter.
  • Check the “Edit Filter Settings” box in the save dialog, else the default delimiter is space, instead of ”,”!
  • Ubuntu Answers "can't change delimiter when a file is already saved as csv".
  • *If the delimiter is the default ”,” ,
    • At “Add Data”, Splunk accepts data with fieldnames containing spaces ( e.g. “Number of Persons” ).
    • At “Search & Reporting / Pivot”, “Fields - Which fields would yo like to use as a Data Model”, such data fields with fieldnames containing spaces may be selected for use as a Data Model.
    • But at “New Pivot”, “Add X-Axis” and “Add Y-Axis”, such data fields with fieldnames containing spaces are not available as fields .
    • So far, I didn´t test if data values may not contain spaces ( e.g. “One Person” ), too.
    • Statistical software as Splunk expects high data quality as available with computer-generated data.
      • If your data values are manually created, and then even have non-numeric data fields ( e.g. for the x-axis “One Person”, “To Persons”, “Three Persons, “More than three Persons” - the last one can´t even be transformed to numeric value ), typing mistakes cause different datasets.
      • So you must check such fields at the “Data Summary” infos of Splunk, if there are just the number of expected values ( e.g. in our example 4 values, and not 5 by a typing mistake like “One Persons” ) for this field.
      • On the other hand, numeric data of numeric fields suitable for y-axis might have repeated values, which might cause a misleading counting at “Data Summary”. Remember the difference between “Count” and “Distinct Count”, with Splunk.
• Splunk expects:
  • Logfiles with a data stamp in each data row. The minimum data stamp is the full date of day, i.e. 2015-11-01.
    • So data which is stored in a single file for each date ( e.g. each day ) is not suitable for Splunk.
    • So data which is not organized by a date stamp ( but e.g. by the zip code of a country ) is not suitable for Splunk.
    • Additionally Splunk offers the date of a file as field “_time”. This doesn´t help much for building a timestamp for data processing, if the file was generated, downloaded or modified manually, or the date of the file is irrelevant due to other reasons.
  • Decriptions of each data table row ( “Field” ), at the top of the file.
  • You may specify at the “Source Type” configuration, that the field names are on a certain line number.
  • However, there is no option to define a final line. So data garbage at the end of the file might disturb the data processing. Especially it would be hard or impossible to put 2 different data sources in one physical file, defined by a single “Source Type”.
• Microsoft Developers Network "C Run-Time Error R6034".
  • Error message when installing Splunk on a Win8.1, 32-bit, where “Python X,Y” is already installed:

    Microsoft Visual C++ Runtime Library
    Runtime Error!
    Program C:\Program Files\Splunk\bin\Python.EXE
    R6034
    An application has made an attempt to load the C runtime library incorrectly.
    Please contact the application's support team for more information.
    

  • Splunk Answers "Splunk services won't start after install error".
  • Splunk Answers "Why am I getting error "R6034 An application has made an attempt to load the C runtime library incorrectly." during a Splunk 6.3 installation?".
  • Splunk Answers "Splunk error - runtime error R6034" - “I uninstalled ActiveState's Python, and now splunkd starts right up”.
• EN.Wikipedia "Unix time", DE.Wikipedia "Unixzeit".
• EN.Wikipedia "Splunk", DE.Wikipedia "Splunk" - “The freeware version is limited to 500 MB of data a day, and lacks some features of the Enterprise license edition”.

• Forbes Tech "Rocana Vs. Splunk: IT Operations Management Battle Of Words", 2015.