Databricks logs collection with Azure Monitor at a Workspace Scale

Databricks is an optimized data analytics platform based on Apache Spark. Monitoring Databricks plateform is crucial to ensure data quality, job performance, and security issues by limiting access to production workspaces.

Spark application metrics, logs, and events produced by a Databricks workspace can be customized, sent, and centralized to various monitoring platforms including Azure Monitor Logs. This tool, formerly called Log Analytics by Microsoft, is an Azure cloud service integrated into Azure Monitor that collects and stores logs from cloud and on-premises environments. It provide a mean for querying logs from data collected using a read-only query language named “Kusto”, for building “Workbooks” dashboards and setting up alerts on identified patterns.

This article focus on automating the export of Databricks logs to a Log Analytics workspace by using the Spark-monitoring library at a workspace scale.

Overview of Databricks log sending

Overview of Spark-monitoring library

This section is an overview of the architecture. More detailed information and the associated source code are provided further down in the article.

Spark-monitoring is a Microsoft Open Source project to export Databricks logs at a cluster level. Once downloaded, the library is locally built with Docker or Maven according to the Databricks Runtime version of the cluster to configure (Spark and Scala versions). The build of the library generates two jar files:

  • spark-listeners_$spark-version_$scala_version-$version: collects data from a running cluster;
  • spark-listeners-loganalytics_$spark-version_$scala_version-$version: extends spark-listeners by collecting data, connecting to a Log Analytics workspace, parsing and sending logs via Data Collector API

In the documentation, once the jars are

Read more
Samsung 32″ 4K curved monitor UR590C (dual monitor review)

In monitor land, there is a saying – you can never have enough space. The Samsung 32″ 4K curved monitor UR590C is enough, well a pair is – for now.

In COVID times, the Officeworks shelves are stripped bare of any work-from-home tech. Fortunately, Samsung makes most of this monitor – the panel and electronics, so you are more likely to both see it in the shop, and if it is out-of-stock, there will be shorter delivery times.

More and more, GadgetGuy is required to test computers, laptops, and docks with 4K monitors – in fact, a standard test now is dual 4K monitors. Samsung kindly agreed to loan us a pair for the testbed.

Off to Samsung’s monitor webpage to explore

Our requirements included a true 3840 x 2160 resolution, 16:9 format and preferably from 30-34″. We also had limited space – a 1200x660mm desk.

We started at Samsung’s monitor webpage to find something suitable. And in doing so went through the lot – Smart, Gaming, High Res, Curved, Flat, and Business. It has a handy filter section that allows you to set screen size, resolution, and more to narrow down the search.

  • Smart M-series is interesting, and we will be reviewing these shortly. These have a Tizen Samsung TV OS to ‘Do it all’ smart monitor. But mainly, being flat ruled them out.
  • Gaming G-series looked awesome but did not have a 4K in the size we were looking for
  • Hi-Res is where we found the Samsung 32″
Read more