Databricks logs collection with Azure Monitor at a Workspace Scale

Databricks is an optimized data analytics platform based on Apache Spark. Monitoring Databricks plateform is crucial to ensure data quality, job performance, and security issues by limiting access to production workspaces.

Spark application metrics, logs, and events produced by a Databricks workspace can be customized, sent, and centralized to various monitoring platforms including Azure Monitor Logs. This tool, formerly called Log Analytics by Microsoft, is an Azure cloud service integrated into Azure Monitor that collects and stores logs from cloud and on-premises environments. It provide a mean for querying logs from data collected using a read-only query language named “Kusto”, for building “Workbooks” dashboards and setting up alerts on identified patterns.

This article focus on automating the export of Databricks logs to a Log Analytics workspace by using the Spark-monitoring library at a workspace scale.

Overview of Databricks log sending


Overview of Spark-monitoring library

This section is an overview of the architecture. More detailed information and the associated source code are provided further down in the article.

Spark-monitoring is a Microsoft Open Source project to export Databricks logs at a cluster level. Once downloaded, the library is locally built with Docker or Maven according to the Databricks Runtime version of the cluster to configure (Spark and Scala versions). The build of the library generates two jar files:

  • spark-listeners_$spark-version_$scala_version-$version: collects data from a running cluster;
  • spark-listeners-loganalytics_$spark-version_$scala_version-$version: extends spark-listeners by collecting data, connecting to a Log Analytics workspace, parsing and sending logs via Data Collector API

In the documentation, once the jars are

Read more
A Collection Of Websites That Look Like Desktops

World wide web design has come a very long way because those halcyon days of Web 1.. There are a great deal of regulations about how to make a clean up and successful web page, but in some cases it’s extra pleasurable to throw them out and just be artistic alternatively. In that vein, [Simone] has curated a great collection of web-sites that emulate the laptop or computer desktop expertise on line.

The collection’s web site extremely significantly fits this concept. On viewing, an Award BIOS display flashes up. From there, we get a Home windows 95-like interface complete of backlinks to other web-sites that emulate a computer system desktop layout. There is even a 3D screensaver that pops up if you mouse absent for as well very long.

There’s Browso.application, which semi-correctly tells you details about your personal computer in a theme reminiscent of MacOS 9. Meanwhile, clicking on “It Is As if You Ended up Executing Work” will just take you to a unusual sport that is compelling in its replication of place of work banality. Nightwave Plaza presents beautiful vaporwave vibes, even though many others simply just intend to faithfully recreate numerous OSes in a browser window.

It’s a entertaining collection of internet websites that go from the weirdly afamiliar to downright amazing recreations of former realities. It’s unquestionably pleasurable to simply click about for a while and see what is out there. We do really like some superior web ephemera all over these pieces!

Read more