Table of Contents
[hemmerling] Data Processing 5/8 - Business Intelligence, Data Mining
Related pages:
Organizations
- DING e.V. - Deutschsprachige Informatica Nutzer Gruppe ( formerly: “Deutschsprachige Informatica Nutzergruppe” ).
- The Data Warehouse Institute ( TDWI ) US, The Data Warehouse Institute ( TDWI ) Europe / Germany.
-
Conferences, Seminars, Trainings
Germany
- Data Modeling Zone in Hannover ( 2013-09-23 - 2013-09-24 ).
-
-
-
- The Data Warehouse Institute ( TDWI ) Roundtable.
- TDWI Konferenz, München.
- 2016.
- TDWI Konferenz 2016 - I joined 2016-06-21 the Seminar “Self-Service Analytics mit Qlik” ( “Self-Service Analytics with Qlik” ) by Digital Ratio GmbH.
- 2018.
- TDWI Konferenz 2018 - I joined the full conference, with a sponsored ticket
.
- 2020.
- The online conference TDWI Virtual, 2020-06-23 - 2023-06-24 - “Die TDWI Virtual ist die kostenfreie Online-Konferenz für Data, BI und Analytics”.
- 2021.
- The online conference TDWI München 2021, 2021-06-21 - 2021-06-23 - I joined the free seminars of the conference.
- 2022.
- The conference with free tracks TDWI München 2022.
- Conference app.
- username: Your E-Mail, initial password: “tdwi2018”.
-
- MicroStrategy Symposium - Munich in München, 2018-10-18.
- TDWI Young Guns.
- Events.
- The free online barcamp TDWI "Oops - we do it again - Data & Analytics BarCamp Vol. 4", 2021-09-16.
- Traveling Salesman.
- The free online barcamp TDWI "Irren ist menschlich – Mach ‘was draus Vol. 2 | Data & Analytics F*ckUp Night", 2022-02-10, 18:00.
-
- 2021.
- 2022.
- 2023-02-24 - 2023-02-25.
- “Global Power Platform Bootcamp 2023” - “Am 24./25. Februar findet das weltweite Global Power Platform Bootcamp 2023 statt”.
- Power Platform User Group Hannover.
- Power Platform User Group Hamburg.
-
-
- Full-day online training DatenPioniere GmbH "Microsoft Power BI Training - Wir machen Euch fit", 2023-02-03, 2023-03-03, 08:30-16:30. I joined.
- “Kostenfreie Anmeldung. Werde mit uns zum Power BI Champion und melde Dich heute noch zur kostenfreien Basis Schulung an”.
- “Jeden Monat bietet Dir Niklas ein kostenfreies Tagestraining an. Wir sind davon überzeugt, dass mehr gute Entscheidungen in deinem Unternehmen getroffen werden, wenn mehr Mitarbeitende Power BI beherrschen. Melde dich heute noch an”.
- Certificate VirtualBadge "Power BI Champion Kurs by DatenPioniere GmbH - Rolf Hemmerling", 2023-02-03.
- As of 2023-01:
- Power BI Pro doesn't provide machine learning / artificial intelligence functions
. - Power BI Premium provides machine learning / artificial intelligence functions.
- Starscheme: Separation of fact table and dimension table.
- Fact table ( FACT_xxx, F_xxx ): German w-questions ( who, what, when, amount, rebate.. ), what is relevant for a financial transaction ( buy / sell / contract ).
- Dimension table ( DIM_xxx, D_xxx ): Product ID, product category, country of customer.
- If a customer buys 1000x the same good, you don't want all the 1000 data items in the dimension table.
- Spreadsheet Cell vs. Calculated column vs. Measure.
- In opposite to spreadsheets, BI tools are not based on the “cell” principle, you can just address columns, not single cells .
- Calculated columns require filespace, Measures don't require filespace, as they are calculated in-memory.
- Calculations
Umsatz = 'F_Order Details'[Unit Price] * 'F_Order Details'[Quantity] *(1-'F_Order Details'[Discount]) Einkaufskosten = 'F_Order Details'[Quantity] * RELATED(D_Products[UnitCost]) Gewinn = 'F_Order Details'[Umsatz] - 'F_Order Details'[Einkaufskosten] Datumstabelle = VAR ErsteErfassung = Min(F_Orders[Order Date]) VAR Kalender = CALENDAR(Date(Year(ErsteErfassung),1,1), Date(Year(Now())+1,12,31)) return ADDCOLUMNS( Kalender, "Jahr", Year([Date]), "JahrQuartal", FORMAT([Date], "yyyy-Q"), "JahrMonat", FORMAT([Date],"yyyy-mm"), "JahrKW", COMBINEVALUES("-", Year([Date]), FORMAT(WEEKNUM([Date],21),"00")), "Quartal", FORMAT([Date], "\QQ"), "Kalenderwoche", WEEKNUM([Date], 21), "Monat", FORMAT([Date], "mmm"), "Monat#", Month([Date]), "Wochentag", FORMAT([Date], "ddd"), "Wochentag#", Weekday([Date], 2) )
- Important settings:
- “File / Options and settings / Options” - “Global / Data Load”.
- ”[_] Time intelligence”.
- “File / Options and settings / Options” - “Current File / Data Load”.
- ”[_] Time intelligence”.
- ”[_] Import relationships from data soruces on first load”.
- ”[_] Autodetect new relationships after data is loaded”.
- Drilldown fields “Category Name, County aus Customers, Last Name aus Employes”.
- After creating a Measure move it to a “Card” ( “123” ) visualisation!
Umsatz Measure = Sum('F_Order Details'[Umsatz]) ! Umsatz Adams I = CALCULATE( Sum('F_Order Details'[Umsatz]), D_Employees[Last Name] = "Adams" ) Umsatz Adams in Deutschland = Calculate( Sum('F_Order Details'[Umsatz]), D_Employees[Last Name] = "Adams", D_Customers[Country] = "Germany" ) Umsatz Adams und Buchanan = CALCULATE( [Umsatz Measure], D_Employees[Last Name] = "Adams" || D_Employees[Last Name] = "Buchanan" ) Umsatz Adams II = Calculate( Sum('F_Order Details'[Umsatz]), D_Employees[Employee ID] = 4 //4 ist die ID für Adams ) Gesamtumsatz = CALCULATE( [Umsatz Measure], All('F_Order Details') Anteil Umsatz = DIVIDE([Umsatz Measure],[Gesamtumsatz]) Umsatz nach Shipped Date = Calculate( Sum('F_Order Details'[Umsatz]), USERELATIONSHIP(Datumstabelle[Date], F_Orders[Shipped Date]) )
- The free event DatenPioniere "BI Arena remote",
BI Arena by DatenPioniere ( bi-arena.de ).- 2023-09-12, 08:30 - 14:15 - Virtual event. I joined.
- 2024-12-10, 9:00 - 13:00 - Virtual event.
- The free event DatenPioniere "BI Arena auf Schalke" at Veltins Arena, Rudi-Assauer-Platz 1, 45891 Gelsenkirchen, 2025-01-29, 8:00 – 14:30.
- I applied, but was rejected: “Leider sind die Plätze vor Ort rar, daher sind diese für Endkunden reserviert”.
-
- “M365 Summits”.
- Online events by BI or DIE Self Service.
- Podcast series BI or DIE Self Service "Podcast".
- Events are linked with banners, on the title page:
- BI or DIE "Women in Data", 2023-03-08.
- BI or DIE "Data Talk", 2023-03-31.
- “BI or DIE “Levelup 2022”, 2023-04-24 - 2023-04-28 ⇒ YouTube.
- Free conferences and expositions Tech Show Frankfurt / Big Data & AI World, Cloud Expo Europe Frankfurt / Data Centre World in Frankfurt, 2023-05-10 - 2023-05-11.
-
-
- Event ”#datatalk congress”, usually in the Ruhrgebiet area, NRW.
- ”#datatalk congress 2015”.
- ”#datatalk congress 2016”.
- ”#datatalk congress 2017”.
- ”#datatalk congress 2018”.
- ”#datatalk congress 2019”.
- ”
#datatalk congress 2020” - Canceled.
- EVACO GmbH "Entscheidend ist die passende Strategie – EVACO #datatalk 2022" - “Weitere Informationen zum Event der EVACO am 18. Oktober 2022 sowie die kostenfreie Anmeldung finden Sie auf unserer #datatalk congress Eventseite”.
- EVACO GmbH "#datatalk congress 2023", 2023-09-26. I joined.
- “SAVE THE DATE #datatalk congress 2023 Di, 26.09.2023 Oktogon, Zeche Zollverein Essen” ( advertised on EVACO GmbH "Entscheidend ist die passende Strategie – EVACO #datatalk 2022" ).
- Workshops:
- “DataRobot Workshop” - Own notebook computer required ( “Sie brauchen nur etwas Entdeckergeist und einen internetfähigen Laptop (als Browser empfehlen wir Google Chrome)” ).
- “Qlik AutoML Workshop” - No own notebook computer required.
- Conference app:
EVACO GmbH "ThoughtSpot Workshop"in Frankfurt, 2024-03-13, 10:00-15:30 - Canceled.- EVACO GmbH "K4 Workshop" in Duisburg, 2023-04-11, 10:00-16:00 - I joined.
- ”#datatalk congress 2024”, 2024-09-12. I was rejected.
- 2024-08-22: “Ihre Registrierung für den datatalk-congress-2024 wurde von ... storniert”, 2024-09-04: “In der Tat ist das Event unseren Kunden und Interessenten vorbehalten. Wir bieten hier unsere Softwarelösungen und Knowhow an. Daher ist für Kunden und Interessenten dieses Event auch kostenfrei. Deswegen müssen wir aber auch etwas genauer schauen und eben andere Berater ausladen. Hierbei spielt die Firmengröße oder Selbstständigkeit keine Rolle. Wir haben bei Ihnen in der Vergangenheit bereits Ausnahmen gemacht – beim #datatalk im letzten Jahr sowie dem K4 Workshop. Mehr ist nun allerdings nicht möglich. Bitte behalten Sie uns in guter Erinnerung” .
-
-
-
- BARC GmbH "BI & Analytics Arena" - Login, Login Code “3744”.
-
- Free conference ElasticON "ElasticON Tour Munich" in München, 2024-11-14.
- Free hands-on workshops.
- “Observability”.
- “Search”.
- “Security.
International
- DataTalks.Club - “Free Data Engineering course!” ( on demand ).
-
-
- The free online conference Data Stack Summit- “Live virtual peer-to-peer sessions”.
- Data Stack Summit 2023, 2023-04-19.
- The free online conference Data Teams Summit ( 2023, 2024 ) & DataOps Unleashed ( 2022 ) - “Live virtual peer-to-peer sessions”.
- Solution Monday "Data Teams Summit 2024", 2024-01-24.
- Data Teams Summit 2024 - Personal Link, 17:00 - 23:00 CET.
- Solution Monday "Data Teams Summit 2023", 2023-01-25 - 2023-01-26.
- Solution Monday "DataOps Unleashed 2022", 2022-02-02 - 2022-02-23.
- The free online conference Data Agility Day, 2021.
-
- Global Data Summit in USA ( 2017, 2019 ).
- The affordable commercial online event Power BI Summit, 2023-03-06 - 2023-03-10.
- The free live virtual event Open Source Data Summit ( OSDS ).
- 2023-11-15.
- 2024-??-??.
Software
Free ETL Software
Free Tools
- The OpenSource apatar - connecting data - “Open Source Data Integration & ETL”.
- apatar - News/Press Releases - “Apatar, the open source leader in data mashup integration”.
-
- Flexible Deployment Options.
- No Coding! Visual Job Designer.
- No Coding! Transformation Mapper.
- The OpenSource CloverETL by Javlin a.s. - “Rapid Data Integration Software and Open Source ETL Tool”.
- SourceForge "CloverETL", SourceForge "CloverETL" - “Rapid data integration framework”.
- The OpenSource Talend Open Studio.
Resources
Free BI Software
Free Data Warehouse Solutions
- “MariaDB AX” → See Data Processing 1/7 - Databases, Webserver, Webservices, SQL, XML.
Free BI Tools
- The OpenSource Apache Superset.
- The free Ataccama "DQ Analyzer" - “Advanced Data Profiling Tool” for Windows.
- The OpenSource BI reporting tool Eclipse BIRT by Actuate Corporation - The BIRT Company.
- BIRT Exchange - “Products and Services for BIRT”, “Get the most out of BIRT”.
- BIRT uses JDBC to access databases. To access an ODBC-only database on Windows, you need a JDBC-ODBC Bridge Driver. The features of the JDBC drivers determine what BIRT can make from the databases, e.g. concerning support of standard and non-standard SQL datatypes ect.
- The fictive company of the sample database is “Classic Models Inc.”.
-
-
-
-
- Free online trainings MicroStrategy Deutschland GmbH - "Kostenloses Online Training".
- The free Oracle Warehouse Builder (OWB) 11g.
- “OWB 11.2 is installed as part of every Oracle Database 11g Release 2 installation”.
- The OWB is linked to a specific version of Oracle Database. So you can´t use OWB 11.2 with Oracle 12 database. As OWB is discontinued, there will be no newer version available for newer Oracle databases ( e.g. Oracle 12,.. ).
- OW2 Consortium - “Leading open source middleware”.
-
- Software:
- “QlikView”.
- The free QlikTech International AB "Download QlikView Personal Edition. Experience the power of Qlik on your personal computer", QlikTech International AB "Seeing is Believing. Downloaden Sie QlikView noch heute!" for Win64 “Qlik lädt Sie dazu ein, sich die QlikView Personal Edition downzuloaden. Diese ist keine Testversion, sondern eine komplette Version unserer leistungsfähigen Business-Discovery-Software”.
- “QlikView 12” is a local server application for Win64 computers, with a rich client applications for Win64.
- “QlikView 11” was the latest edition with both Win64 and Win32 server and client applications.
- “QlikView” is more complex than “Qlik Sense”, i.e. for domain experts with good software programming skills.
-
- Accompanying website QlikView 11 for Developers.
- Application developers need both the “QlikView” application server and rich client.
- Important commands:
- By “File / Edit Script” you may create and modify the query code to access and filter the data of the project.
- By “File / Table Viewer” you may view the table structure of the project.
- With “QlikView”, you must import each single spreadsheet worksheet, e.g if a spreadsheet file consists of 3 worksheets, you must do 3 manual imports.
- “Qlik Sense”.
- The free QlikTech International AB "Download Qlik Sense Desktop. Experience the power of Qlik on your personal computer", QlikTech International AB "Download von Qlik Sense Desktop. Kostenfreie Self-Service Datenvisualisierung und Discovery - vertrauen Sie Ihrer Intuition" for Win32 or Win64 - “Qlik Sense Desktop ist eine kostenfreie Windows Desktop Applikation, mit der Einzelpersonen über Drag-and-Drop interaktive Datenvisualierungen, Reports und Dashboards aus verschiedensten Datenquellen erstellen und teilen können. Gewinnen Sie in nur 5 Minuten mehr Einsichten in Ihre Daten!”.
- “Qlik Sense” is a local server application for Win64 computers. You may operate with the desktop client “Qlik Sense Desktop” - which is a modified browser - or may use a standard Internet browser instead, by calling URLs:
- Application developers just need the “Qlik Sense” (enterprise) application server, on which the application is published.
- “Qlik Sense” is more simple than “QlikView”, i.e. for domain experts without software programming skills.
- I was told by sales staff in 2016, that new customers prefer to order “Qlik Sense” prior to “QlikView”.
- The online service QlikTech International AB "Join Qlik Sense Cloud. Experience the power of Qlik on your personal computer" with free entry-level account offer.
- With “Qlik Sense”, you may import a spreadsheet at once, e.g if a spreadsheet file consists of 3 worksheets, you must just do one import.
- Experts told me about filtering:
- Filters work as associative logic in context.
- A number of filters may may be applied to an Qlik application.
- If you apply data filtering to the Qlik data model, it is applied to all Qlik applications which use it.
- If you apply data filtering to a Qlik application, it is applied to the single Qlik application.
- Design rules for dashboards:
- Simple rule:
- Left side = “Filter”.
- Right side = “Information”.
- Resources:
- Qlik Community "Guide to Dashboard Design.pdf" 2014 → PDF.
- The commercial Java application Pentaho Corporation, Pentaho Corporation - Germany.
- it-novum GmbH - German service partner.
-
- The free Pentaho BI Platform / Server. The Pentaho BI platform provides architecture and infrastructure required to build solutions to business intelligence (BI) problems = “Pentaho BI Platform Community Edition (CE)”.
- The OpenSource Pentaho Reporting Project. A collection of primarily focused on the creation, generation and distribution of rich and sophisticated reporting from all sources of information = “Pentaho Reporting Community Edition (CE)”.
- The OpenSource Pentaho Kettle Project. A powerful, metadata-driven ETL tool designed to bridge the gap between business and IT; Turning company's data into increased profits = “Pentaho Data Integration Community Edition” ( PDI CE ).
- The OpenSource Penatho Modrian Project. Open source analysis OLAP server written in Java. Enabling interactive analysis of very large datasets stored in SQL databases without writing SQL - “Pentaho Analysis Services Community Edition”.
- The OpenSource Pentaho Weka Project. A comprehensive set of tools for machine learning and data mining used to help you understand the business better through predictive analysis - “Pentaho Data Mining Community Edition (CE)”.
- The OpenSource Java application SourceForge "Pentaho Business Analytics", SourceForge "Pentaho Business Analytics".
- EN.Wikipedia "Pentaho", DE.Wikipedia "Pentaho" - “Pentaho Data Mining used the Waikato Environment for Knowledge Analysis (Weka) to search data for patterns”.
- Third-party applications.
- The commercial SAS software.
- The free SAS University Edition - “Kostenlose Statistik Software”.
- The OpenSource commandline application VisiData - “An interactive multitool for tabular data. It combines the clarity of a spreadsheet, the efficiency of the terminal, and the power of Python, into a lightweight utility which can handle millions of rows with ease”.
- The OpenSource application Weka 3: Data Mining Software in Java.
- The announced “XDEV BI Suite” of “XDEV 4” ?!
Free BI Frameworks
- The OpenSource framwork CERN "ROOT".
- ROOT software script can also be executed by the C++-Interpreter CINT.
- The OpenSource Python library “pandas” → See Python.
Data Cleanup
- The OpenSource OpenRefine, GitHub "OpenRefine" ( formerly: “Google Refine” ) - “A powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data”.
Machine Learning
Important Machine Learning Tools according to "Developer Economics - National Trends Survey", 2016-04
- IBM Watson
- BigML
- SAS Enterprise Miner
- R platform
- Python machine learning libraries
- Google Prediction API
- WEKA Machine Learning Workbench
- Microsoft Azure Machine Learning
- AWS Machine Learning
- TensorFlow
- PredictionIO
- MATLAB or Octave
- DMTK
- IBM SPSS Modeler
- Apache Spark
- RapidMiner
Tools
- The OpenSource library Tensorflow, GitHub "tensorflow" - “TensorFlow is an Open Source Software Library for Machine Intelligence”, “Open source software library for numerical computation using data flow graphs”.
- GitHub "google/skflow" - “Simplified interface for TensorFlow (mimicking Scikit Learn)”.
Resources
Microsoft BI Tools
The free standalone Microsoft Power BI
The Tool
-
- The free dashboard tool “Power BI Desktop for Windows” for Win7, Win8 - “Analytics tools at your fingertips”.
-
- Blog Microsoft Community Blog.
Online Services
- “You've selected Microsoft Power BI. Let's get you started. XXX looks like a personal email address. Enter your work address so we can connect you with others in your company. And don't worry. We won't share your address with anyone. Sign in with a work email address”.
- The publishing service “Power BI”, just for users of the commercial “Office 365” service?! - “The easy way to see your important data in one place. With a few clicks, connect to data from applications you use and get started with pre-built dashboards from experts”.
Downloads
-
- Just for users of the commercial “Office 365” service?!
- Google Play "Microsoft Power BI" for Android smartphones, “Requires Android 4.4 and up”.
- Just for users of the commercial “Office 365” service?!
- But Google Play refuses installation “Your Device isn´t compatible with this version” on simulator Bluestacks with “Android 4.4.2”.
Resources
- “Microsoft Power BI” is a dashboard tool, not a reporting tool!
- It is powered by “Microsoft SQL Server Analysis Services (SSAS)”.
- It is now easy to catch data from HTML pages ( e.g. data tables of Wikipedia ).
- Data visualisation:
- Built-in graphics ( by a graphics engine similar / identical to that known from Micrsoft Excel, Microsoft SQL Server Reporting Services and Microsoft SQL Server Analysis Services ).
- There is an API by which third parties may offer own visualisations.
- Indeed with Microsoft Excel 2016, you may create the same dashboards, but you can´t publish it on the web.
- You may save “Microsoft Power BI Desktop” as in single ”.pbix” files. Experts told me, that for saving a project, the software needs 3 times of the RAM memory as needed for the loaded data. So it might be that you may load big data, but are not able to save it. In general, this might not be a problem with the 32-bit edition ( on Windows PCs with 2 or 3 GByte of RAM ), but also with the 64-bit edition ( i.e. a project may be saved on a Win64 computer with 16 GB RAM, but not on a Win64 computer with 4 GB RAM ).
- You might create test cases ( e.g. at business intelligence trainings ) with “Microsoft Power BI”, as alternative to NBI.
- NBi, GitHub "Seddryck/NBi", CodePlex "NBI" - “Testing framework for Microsoft Business Intelligence platform”.
- “Microsoft Power BI Desktop” also supports the “M” and “DAX” query languages, by the “Advanced Editor” ( “Home / Edit Queries ( = Query Editor ) / Advanced Editor” ).
- If you install “Microsoft R” prior to “Microsoft Power BI Desktop”, you may use “R” for data processing and graphical reporting → See Mathematical Engineering.
- I was told by experts, in 2016-03:
- Now legacy versions of “Microsoft Power BI Desktop” were able to process 10.000 datasets for visualisation.
- Current versions of “Microsoft Power BI Desktop” were able to process 10.0000 datasets for visualisation.
- With “R” called by “Microsoft Power BI Desktop”, you may visualize 150.000 datasets.
Commercial Editions of Microsoft SQL Server with BI Support
Power BI Gateway - Personal
The Tool
Documentation
Concept
- To get a BI report, connect with a client ( Internet browser, smartphone app,..) to the “Microsoft Power BI” cloudservice. The basic “Power BI” account is free.
- By this, you may access any cloud document.
- The cloud servce may access a registered onpremise Windows server at your ( the datacenter of your ) company, if the computer runs a free “Power BI Gateway - Personal” service, by a VPN connection.
- Experts told me, that the free VPN service shipped with Windows is not under development for some time. Therefore it isn´t very advisable to use it in production environments.
Resources
Free Addons for the commercial Microsoft Excel
Tools
-
- Powerful BI Now Available in Excel 2013:
- xVelocity In-memory Analytics.
- Quick Explore, Quick Analysis and Flash Fill.
- Power View.
- Power Pivot.
- Microsoft Power Query for Excel ( formerly: Microsoft Data Explorer for Excel ).
-
- Blog Microsoft MSDN "Power BI" ( formerly: Microsoft MSDN Blog “Data Explorer Microsoft 'Data Explorer' Preview for Excel” ).
Query and Data Modeling Languages
Azure Log Analytics Query Language
- See on this page.
Analysis Services Scripting Language ( ASS )
- EN.Wikipedia "XML for Analysis" ( XMLA ).
Data Analysis Expressions ( DAX ) - Data Modeling Language
- Tbe OpenSource software SQLBI "DAX Studio", CodePlex "DAX Studio" for Windows.
- Experts told me, that you may connect from “DAX Studio” to a (running) PowerBI model.
"M" Language ( "Power Query Formula Language" / Microsoft Power Query for Excel Formula Language" )
MultiDimensional eXpressions ( MDX )
- Tbe OpenSource software ]SQLBI "MDX Studio"], [[http://daxstudio.codeplex.com/|CodePlex "MDX Studio v2" for Windows.
- Microsoft.
Tabular Modeling Scripting Language ( TMSL )
- Experts told me, that “Tabular” is like “PowerBI” / “QlickView” / “Tableau” but without graphical user interface and graphical reporting. Instead, “Tabular” may be accessed by any tool by DAX & MDX query language.
-
Datazen
-
- “Microsoft Acquires Datazen”, 2015-04-14.
- The functionality of Datazen is included in “SSRS 2016” for
- Windows 10 Desktop ( but not Windows Phone! ).
- Android.
- iPhone.
Data Science Platform by Joshua Görner
- The commercial online course Beyond Jupyter - Take your data science skills to the next level.
- Beyond Jupyter "Legal Disclosure" with E-Mail address of “Joshua Görner”.
-
- Free conference SIGS DATACOM GmbH "Code Days" as part of the conference OOP 2018 at the ICM - Internationales Congress Center München, München ( 2018-02-17 - 2018-02-18 ).
- SIGS DATACOM GmbH "Code Days" - "Programm-Details", 2018-02-18 - “Joshua Görner: Beyond Jupyter Notebooks: Build your own Data Science Platform in less than 1 hour”.
- On the TDWI 2018 conference in München, Germany, Joshua Görner ( TDWI Konferenz 2018, Sprecherdetails "Joshua Görner" ) was speaker. I joined his seminar “Beyond Jupyter Notebooks: Build yoru own Data Science Platform in less than 1 hour”.
- Suggested “Data Science Platform”:
- Jupyther.
- See Python 1/10.
- Superset.
- PostgreSQL / PostgREST.
- minio.
- Minio - “Private Cloud Storage. Minio is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. Minio is widely deployed across the world with over 105.1M+ docker pulls”.
- API Star.
- API Star, GitHub "encode/apistar" - “A smart Web API framework, for Python 3”.
- Docker.
- Joshua Görner suggested to use the Python web framework “Flask” to build datascience applications. “Flask” is providing easily “Functions as a Service” for datascience applications. See Python 10/10 - Web Application Frameworks.
"TWDE Datalab" by ThoughtWorks
- GitHub "ThoughtWorksInc/twde-datalab" - “Onboarding to data science by ThoughtWorks”.
- The speech “Mi 9.1: A Hands-on Introduction to Your First Data Science Project” by A. Wider, E. Grasmeder at the conference TDWI Konferenz 2018 by TDWI Germany e.V., 2018-06-25 - 2018-06-27, introduced me to the “TWDE Datalab”.
Some other commercial BI Software
- INTEC International GmbH - “Monitoring von Geschäftsdaten mit dem BigData Monitor”.
- The graphical data organisation scheme is:
- Size of a rectangle.
- 3 colors ( transition from red to white and then to blue ).
- Label for each rectangle.
- This is an adoption of the product idea of Technische Universität Eindhoven "SequoiaView".
-
-
- The commercial software TIBCO Software Inc. "Spotfire".
- OnTheHub "Download Spotfire S+ 8.0 for Windows Student Edition - Free “TIBCO Spotfire S+ 8.0 (1 Year License)”.
- it-production.com -Das Online-Magazin für erfolgreiche Produktion "Zwischen Top Floor und Shop Floor 'Die Zwei-Sekunden-Lücke nutzen'" - “Die Frage, ob sich Business Intelligence-Systeme für Auswertungen im Produktionsumfeld genauso eignen wie für Forecast-Aufgaben für das Management..”.
Services
BigML
- The online cloud based predictive analysis service BigML - “Machine Learning for everyone. Easily add data-driven decisions and predictive power to your company”, “NOW FREE. Unlimited tasks ( up to 16 MB/task )”.
EdgeImpulse Inc.
-
- EdgeImpulse Inc. "Pricing" - Free developer account ( “Developer. For developers looking to deploy innovative ML on any edge device = Free” ).
-
- EdgeImpulse Inc. Docs "Development Boards" - “There is a list of development boards that are fully supported by Edge Impulse”.
- Onsite event EdgeImpulse Inc. "Presenting Edge ML Series. Frankfurt, Amsterdam, Milan, Paris, London, Munich".
- 27-28 April Frankfurt - (27.4.2022 12:00-18:00 and 28.4.2022 8:30-13:00).
- 3 May Amsterdam - (3.5.2022 all day), 08:30-18:00 ( 17:00 ) ⇒ I applied and was accepted.
- Renaissance Amsterdam Hotel, Kattengat 1, 1012 SZ Amsterdam, Netherlands.
- Smartphone app:
- Deutsche Welle "Einreise- und Corona-Regeln: Niederlande", 2022-04-20 - “Einreiseregeln. Die Niederlande gilt aus deutscher Sicht nicht mehr als Hochrisiko-Gebiet. Für Reiserückkehrer aus den Niederlanden entfällt damit die Digitale Einreiseanmeldung, für Ungeimpfte ist keine Quarantäne mehr nötig”, “Seit dem 23. März ist die Einreise in die Niederlande ohne Nachweispflicht möglich. Im Land wurden fast alle Corona-Regeln aufgehoben”.
- 5-6 July Munich - (5.7.2022 12:00-18:00 and 6.7.2022 8:30-13:00).
Google Colab ( Colaboratory )
- The free online service Google Colab ( Colaboratory ).
- A free Jupyter Notebook server, most packages of Anaconda are installed, but not 100% compatible ( different version of Python packages, few missing packages ).
- User experience of Google Drive users ( so you must be logged into Google Drive by your Google account ):
- Create new Jupyter cells.
- Button ”+Code” and menu option “Insert / Code cell” create a new code cell, while ”+Text” and menu option “Insert / Text cell” create a new text cell, at the point of the Jupyter notebook, where your cursor is, i.e. where you have hit your mouse by mouse-leftlick.
- In local Jupyter notebooks, there is a button of the extra button bar to provide this. This button bar is not available with Google Colab.
- You may create a copy of a Jupyter notebook put on Google Docs, by “File / Save a copy in Drive”.
- But its most easy to transfer additional data files by downloading them on your local terminal ( PC with browser ), and to upload it into the copied Jupyter Notebook.
- You might use “Files / Mount Drive” to create the code
from google.colab import drive drive.mount('/content/drive')
Google Looker
- The commercial online BI tool Google Cloud "Looker for Business Intelligence".
Microsoft KUSTO
The Service
- Microsoft "Azure Kusto" - “Kusto is a log analytics cloud platform optimized for ad-hoc big data queries”.
- Microsoft KUSTO "Documentation", Microsoft KUSTO "Documentation, Microsoft KUSTO "Documentation - Not accessable with simple free Microsoft account.
Azure Log Analytics Query Language
Resources
SAP HANA
The Service
- The online cloud based service SAP HANA, with free developer account.
- SAP HANA Cloud Platform - Login for the free developer account.
-
- SAP HANA Cloud Platform Help "Signing Up for a Developer Account" - “The name of your new developer account contains your user ID ( e.g. 'p0123456789' ) and the suffix trial ( e.g. 'p0123456789trial' )”. However, you still have to login with your SAP ID or E-Mail address, not your developer ID.
-
-
- SAP HANA Developer Center - Signup here for a free developer edition account of SAP HANA in the cloud.
- As of 2016-02, you may install developer editions on SAP HANA Cloud Platform, Microsoft Azure Cloud, Amazon Web Services only. Experts expect, that after 2016-05 there will be a downloadable developer edition and / or a entry-level commercial edition of SAP HANA.
Education
- Webcasts.
-
Events
-
- Software to install for the ”SAP Codejam (SAP HANA with IoT)”:
- Node.js.
- ”Node.js 0.12.10”, for use with “Tessel1” boards. See → ARM Embedded Systems ( ARM7, ARM8, ARM9, ARM11, Cortex ).
- NodeJS for use with Texas Instruments Sensortag. See → Wireless Networks.
- Browser Google Chrome.
- HTML/JavaScript editor.
Resources
Some other commercial BI Tools
- Google Cloud “Looker for Business Intelligence”.
-
- EVACO GmbH - German partner of K4.
- By the help of a “Microsoft Teams” extension, K4 is available on mobile devices ( Android, iPhone ) too.
- Field service employees don't have to learn a BI tool like Qlik, but “just” operate teams. The data can then be exported to K4 via the K4 Extension for Teams.
- Qlik.
- Qlik Extensions:
- K4.
- Experts told me, that Qlik extensions are not available on mobile devices ( Android, iphone ).
Complex event processing, Streaming Analytics
Realtime Data Warehouse
FIWARE "Orion ( Context Broker )"
- See FIWARE 1/2.
Google Cloud Stream Analytics Solution
Microsoft Technologies for Realtime Data Warehouses
Microsoft Azure Stream Analytics
-
-
- “Azure Event Hubs provides highly scalable publish-subscribe event ingestors. An event hub can collect millions of events per second, so that you can process and analyze the massive amounts of data produced by your connected devices and applications”.
- “Together, Event Hubs and Stream Analytics provide an end-to-end solution for real-time analytics. Event Hubs lets you feed events into Azure in real-time, and Stream Analytics jobs can process those events in real-time”.
- “For example, you can send web clicks, sensor readings, or online log events to Event Hubs. You can then create Stream Analytics jobs to use Event Hubs as the input data streams for real-time filtering, aggregating, and correlation”.
-
- “In a real-world scenario, you could have hundreds of these sensors generating events as a stream. Ideally, a gateway device would run code to push these events to Azure Event Hubs or Azure IoT Hubs. Your Stream Analytics job would ingest these events from Event Hubs and run real-time analytics queries against the streams. Then, you could send the results to one of the supported outputs”
-
- GitHub "Microsoft Azure Stream Analytics" - “This repository contains data samples and data generators used for Azure Stream Analytics tutorials”.
Microsoft SQL Server - StreamInsight
- Microsoft Library "Columnstore Indexes" - Additional technology for “Realtime Data Warehouses”.
Microsoft KUSTO
- See on this page.
Resources
- “Data Ingest” ( German: “etwas (
Nahrung) zu sich nehmen”, “Daten einfügen” ) - If you process data - by machine learning, by statistical methods -, the resulting data is added ( “ingested” ) to the original data. The term “insert” is not used, as it is already used with SQL. -
- “SQLstream: Blaze enables companies to be data-driven in real time. Real-time data can be discovered, analyzed, and aggregated instantly, and delivered as a continuous ingest into Hadoop, data warehouses, and other enterprise systems”.
Data Analysis & Dashboards
- Grafana.
- Kibana.
- The OpenSource plugin Elasticsearch Kibana - “Explore & Visualize Your Data”.
-
- Elastic "Learn, Docs" - "Kibana Dashboard" - “A Kibana dashboard displays a set of saved visualizations in groups that you can arrange freely. You can save a dashboard to share or reload at a later time”.
- Kibana 4 Tutorial - “A web frontend to analyze data held in an elasticsearch cluster”.
- EN.Wikipedia "Kibana" - “Open source data visualization plugin for Elasticsearch”.
Full Text Search
- Elitness.
-
-
- “3.4 The Eliteness Model and BM25. We now re-introduce term frequencies into our model... We suppose that for any document-term pair, there is a hidden property which we refer to as eliteness. This can be interpreted as a form of aboutness: if the term is elite in the document, in some sense the document is about the concept denoted by the term. Now we assume that actual occurrences of the term in the document depend on eliteness, and that there may be an association between eliteness (to the term) and relevance (to the query)”.
-
- EN.Wikipedia "Full text search", DE.Wikipedia "Volltextrecherche" - “Zur schnellen Informationsgewinnung und dem Auffinden aus bekannten wie auch nicht bekannten (aber auf den Medien vorhandenen) Dokumenten wird die Volltextrecherche genutzt. Die Volltextrecherche dient daher dem Auffinden, Entdecken und Extrahieren unbekannter, nicht trivialer und wichtiger Informationen aus großen Mengen von unstrukturierten Texten/Dateien und ist somit auch ein wichtiger Teilbereich des Text Mining. Sie ist eine Sofortlösung, um für eine konkrete Fragestellung auf Systeme wie Dokumentenmanagement und Data-Mining verzichten zu können”.
- Document management → See Content Management Systems ( CMS ).
- Data-Mining → See “Business Intelligence” on this page.
Artificial Intelligence
Webcasts. Webinars
-
- Webcast “Essentials of Business Analytics & Big Data: What Every Service Leader Needs to Know”.
Experts
- AandBI.com powered by Grete DV GmbH ( XING "Markus Alexander Grete" ), Burgdorf - “Implementierung und Schulung KI, BI, Datenschutz, Implementation and Education, Artificial Intelligence, Business Intelligence and Privacy Law”.
Literature
- PDF book TDWI "The Essential Guide to Machine Data", 2019-07.
- PDF book MemSQL, Aileen Nielsen "Cutting Edge Approaches to Time Series Analysis", 2019-08.
Resources
- While commercial editions of OpenSource BI software tools start with 10.000 EUR/a... 50.000 EUR/a for single customers, there are totally different costs, if the customer wants to integrate or bundle the commercial edition of the OpenSource BI software tool with the own software solution to end-customers, as OEM partner.
- I was told by experts, that some business intelligence software tools are not based on multidimensional databases, but SQL, NoSQL,...
- The competition Data Mining Cup.
- Data Science Central - “The Online Recourse for Big Data Practitioners”.
- KDNuggets - “Data Mining Community's Top Resource”.
- Speicherguide "Big Data heißt: Loslassen von strukturierten Daten" - “Polystrukturierte Daten ist ein neuer Begriff, der strukturierte und unstrukturierte Daten sowie die neuerdings massiv auftretenden maschinengenerierten Daten wie Sensor-Daten und Web-Logs mit einbezieht”.
-
- Blog THE SQL Server Blog Spot on the Web "SQLBI - Marco Russo" - “SQLBI is a blog dedicated to building Business Intelligence solutions with SQL Server”.
- Wikipedia.
- EN.Wikipedia "Mitigation", DE.Wikipedia "Folgenminderungsstrategie" - “Information technology: Mitigation, a kind of defense against security issues in computing, as part of vulnerability management”.
Forums, Newsgroups
- Facebook.
-
-
- LinkedIn:
- Microsoft
-
- Microsoft Technet Forums - Questions tagged with "powerbi" - “All Power BI related forums have been moved to http://community.powerbi.com/]. These forums are now locked and read-only. We sincerely apologize for the inconvenience this may have caused you”, 2015-07-17:
- “Power BI Dashboard Preview”.
- “Power BI for Office 365”.
- “Power BI Mobile Apps”.
- “Power BI Q&A”.
-
-
- TIBCO.
- The read-only ancient Forum TIBCO Spotfire Community.
When this document changes ! Site Navigation ( My Business ! My Topics ! Imprint / Contact ! Privacy Policy ! Keyword Index ! Google+ Author "Rolf Hemmerling" ! Google+ Publisher "hemmerling" )
en/bintelligence.html.txt · Last modified: 2025/07/11 20:01 (external edit) · []
