Why is the level of Clojure support important? As an example, consider that Scicloj is mostly focused on cases where your data fits on a single machine. As such, if you need to work with a large dataset it will be necessary to compute on-cluster and extract a smaller result before continuing your data science task locally.

However, without sufficient Clojure support for on-cluster processing, anyone needing that facility for their data science or data engineering task would be forced to reach outside the Clojure ecosystem. That adds complexity in terms of interop, compatibility and overall stack requirements.

With that in mind, let's examine the level of Clojure support for some popular on-cluster data processing tools. For each tool I selected its official Clojure library if one exists, or if not the most popular and well-known community-supported alternative with at least 100 stars and 10 contributors on GitHub. I then used the following criteria against the library to classify it as "supported" or "support unknown":

1. CI/CD build passing
2. Most recent commit less than 12 months ago
3. Most recent release less than 12 months ago
4. Maintainers responded to any issue or question less than 12 months ago
5. Maintainers either accepted or rejected any PR less than 12 months ago

If I couldn't find any such library at all, I classified it as having "no support".

Please note, I don't wish to make any critical judgments based on either the summary analysis above or the more detailed analysis below. The goal is to understand the situation with respect to Clojure support and highlight any gaps, although I suppose I am also inadvertently highlighting the difficulties of maintaining open source software!

Spark Interop with Geni

Geni is the go-to library for Spark interop. Some months back, I was motivated to evaluate the coverage of Spark features. In particular, I wanted to understand what would be involved to support Spark Connect as it would reduce the complexity of computing on-cluster directly from the Clojure REPL.

However, I found a number of issues that would need to be addressed in order to support Spark Connect and Databricks:

1. Problems with the default session.
2. Problems with support for Databricks, although I suspect this is related to point 1.

Also, in general by my criteria the support classification is "support unknown":

1. CI/CD build failing.
2. Version 0.0.42 api docs broken, also affects version 0.0.41
3. No commits since November 2023.
4. No releases since November 2023.
5. No PRs accepted or rejected since November 2023.
6. No response when attempting to contact the author or maintainers.

Kafka Interop with Jackdaw

Jackdaw is the go-to library for Kafka interop. However, by my criteria the support classification is also "support unknown":

1. No commits since August 2024.
2. No releases since December 2023.
3. No PRs accepted or rejected since August 2024. As a further example, here's a PR raised in May 2024 but not yet commented on either way by the maintainers.

Snowflake Interop with a New Clojure API!

Although the Snowpark library has Java and Scala bindings, it doesn't provide anything for Clojure. As such, it's currently not possible to interact with Snowflake using the Clojure way.

To address this gap, I decided to try my hand at creating a Clojure API for Snowflake as part of a broader effort to improve the overall situation regarding Clojure support for popular data tools.

The aim is to validate this approach as a foundation for enabling a wide range of data science or data engineering use cases from the Clojure REPL, in situations where Snowflake is the data warehouse of choice.

The README provides usage examples for all the current features, but I've copied the essential ones here to illustrate the API:

Load Clojure data from local and save to a Snowflake table

(require '[snowpark-clj.core :as sp])

;; Sample data
(def employee-data
  [{:id 1 :name "Alice" :age 25 :department "Engineering" :salary 75000}
   {:id 2 :name "Bob" :age 30 :department "Marketing" :salary 65000}
   {:id 3 :name "Charlie" :age 35 :department "Engineering" :salary 80000}])

;; Create session and save data
(with-open [session (sp/create-session "snowflake.edn")]
  (-> employee-data
      (sp/create-dataframe session)
      (sp/save-as-table "employees" :overwrite)))

Compute over Snowflake table(s) on-cluster and extract results locally

(with-open [session (sp/create-session "snowflake.edn")]
  (let [table-df (sp/table session "employees")]
    (-> table-df
        (sp/filter (sp/gt (sp/col table-df :salary) (sp/lit 70000)))
        (sp/select [:name :salary])
        (sp/collect))))
;; => [{:name "Alice" :salary 75000} {:name "Charlie" :salary 80000}]

As an early-stage proof-of-concept, it only covers the essential parts of the underlying API without being too concerned with performance or completeness. Other more advanced features are noted and planned, pending further elaboration.

I hope you find it useful and I welcome any feedback or contributions!

However, the leading EdTech platforms don't have much support for Clojure so all the potential benefits of both the language and Scicloj are not currently accessible to those communities.

The good news is that I have created a proof-of-concept for using a browser to write, load and evaluate Clojure code running on a remote server using websockets, processing the results for display using the Scicloj notebook library clay.

I don't believe this combination has been achieved before. It is a significant step when you consider that Scicloj is Java/JVM-based on account of the underlying math support, there is no Javascript or ClojureScript implementation and that is likely to remain the case.

This work opens up the possibility for Clojure-based data science on e-learning platforms so that anyone anywhere can learn and experiment with the Scicloj stack.

Here are some examples of new e-learning content that could be unlocked..

Theory:

• Value vs state & functional programming
• Concurrent programming.

Clojure hands-on:

• Interactive programming and structural editing with the REPL
• Data processing with lazy sequences and transducers
• Data science notebooks covering stats, ML or LLM with rendered tabular data and charts.

More recently I gave an update on my progress at the Scicloj Visual Tools #34 meetup, including a live demo:

I hope you can appreciate the opportunity here. I'm happy to give a live demonstration to anyone who's interested!

In a computer program, when values change over time we call this state. This is why we have two different words available to us for differentiating the context: the specific use of “state” instead of “value” signifies to the reader that we are intentionally composing two things, namely value and time.

Each of those concepts are simpler to reason about on their own, but when put together they require much more care. When you hear people saying “state is inherently complex”, this is what they are referring to. This is especially relevant when we are learning about data streaming tools as they need to consider state in many areas, and at scale: windowed aggregations, joins and other stateful operations, not to mention horizontal scaling, memory management through watermarks and checkpoints, fault tolerance and more.

So how best to understand the state management of these tools? Let’s take a look at what some of the most popular options provide to educate and inform users in this regard:

Based on the above, we can see that

• Despite their complexity, these tools are mostly limited to written documentation for educating and informing users on their stateful operations.
• While they all provide unit test facilities, these are intended to test your usage or composition of these operations rather than understanding the operations themselves. One could also argue that unit testing is targeting a later phase of your project than “educate and inform”.
• They are somewhat limited in any checks of their execution plans, and as a result things can slip through the cracks.

With that in mind, consider again that stateful data streaming problems necessarily involve the consideration of time, and as such they are fundamentally a dynamic concern. By contrast, written documentation is of course only static and for that reason I will submit that it is inefficient and inadequate for the intended purpose.

Indeed, I was affected by this issue personally when I ran into trouble with one of these tools. I still don't know if the problem I encountered is due to a misunderstanding of the (20,000 word) documentation or a bug.

So what’s the solution? Well, consider that we learn best by a combination of reading and doing rather than by reading alone. The “doing” is something that happens in real time, and one way to achieve this is by simulation.

In our case, I will define a simulation thus:

A means to observe the effects of a stateful operation, where for the same input as given to the production equivalent, the simulation will give the same output.

Given the stated purpose of a simulator in our case is to educate and inform, I will also add to the definition that it must require zero installation or setup. Further, to reduce costs and complexity it should also require minimal resources and ideally be fully serverless or standalone in operation. Finally, since the goal is to represent stateful operations, it should be capable of representing those in a visual dynamic by using animated forms for example.

I think there’s an opportunity for these tools (or new ones) to provide visual simulators as the primary means of reasoning for their stateful operations, and also as a complement to their existing documentation.

So with that out of the way, if we could build such a simulator what would it look like, how would it work and how could it be built? Here’s a motivational blueprint!

1. Make the simulator available in a web browser.
2. Write the core functions for the streaming solution and its stateful operations in a hosted language that compiles to code that can run in a browser. Then the same code can be used for both a production implementation and the browser-based simulation.
3. Represent unbounded inputs using generators over lazy sequences.
4. Define the execution plan specification and plan validation rules as data. Then, both the code that checks plans against the rules and any written reference guide can parse this same data, avoiding the possibility of inconsistencies.
5. Within the simulator, represent stateful operations as declarative example-based or property-based BDD style given-when-then constructs, with an animated accumulation of results over time.

In conclusion, I hope you can appreciate the benefits that simulations would bring in this space, and I also hope to have suitably motivated other people in the community to take the baton!

Credits.

Igor Garcia for your feedback and advice: thank you 🙏

1. Flink provides an operations playground but it doesn’t specifically cover stateful operations. There’s also a worked example based on fraud detection, but the explanations are 100% written.↩
2. Flink performs semantic checks for jobs defined using the Table API and SQL. However, the pre-execution validation is not exhaustive, and certain subtle errors or issues might only manifest as unexpected behavior or silent failures during runtime.↩
3. Kafka Streams doesn’t have a distinct pre-execution validation phase in the traditional sense. Instead it relies on a combination of static type checking and the TopologyTestDriver as part of a unit testing strategy.↩
4. Spark Structured Streaming has a multi-layered validation process to ensure correctness and feasibility of computations before their execution. The UnsupportedOperationChecker enforces streaming-specific rules during the logical planning stage.↩
5. Apache Storm's pre-execution topology checks focus on structural and configuration validity, and exceptions are thrown to indicate structural problems, configuration errors and authorization failures. Although it provides facilities for programmatically defining and inspecting topology structure and configuration, a dedicated validation API against documented capabilities is absent.↩

It's a good question! Let's start by defining the problem. A collection is defined here as any number of items, and partitioning is defined as the division of said items into n groups of equal size, with the last group possibly containing fewer than n items.

Well that seems simple enough to understand. It's a single task with only one dimension and no dependencies. I wouldn't be surprised if a child could apply this algorithm using real objects without understanding how to code it, so it should be relatively simple for us to implement, right?

Firstly, let's see how it could be done with the three most popular languages ¹ using Stack Overflow data as a guide (captured on 14th Feb):

Python. Lots and lots of ways, based on the most popular post with 1.7m views and 69 answers.

Java. Lots of ways, based on these popular posts with 250k views and 23 answers, 166k views and 18 answers, unless you can use Java 22. ²

Javascript. Lots and lots of ways, based on the most popular post with 1.3m views and 88 answers, unless the environment supports Baseline 2024. ³

In conclusion, for all these more popular languages we have

• Many different ways, with different syntax and different apis.
• The suitability of each depends on the context (small / large, sync / async) and details of the data structure we're working with.
• The availability of each depends on the version, of which there are many.

By contrast, let's look at three much less popular languages ⁴ again consulting Stack Overflow:

Kotlin. Just a few ways, based on the most popular post with 37k views and 6 answers. Although there are 6 answers, our problem can be solved using kotlin.collections/chunked in the standard library from version 1.2 onwards. ⁵

Rust. Just a few ways, based on the most popular post with 6k views and 2 answers. Although there are 2 answers, our problem can be solved using trait.Iterator/array_chunks from the standard library. ⁶

Clojure. I couldn't find a specific Stack Overflow question for our case, but no matter: the top Google result takes me directly to partition and from there to partition-all. ⁷

There seems to be a clear difference here. Despite the expected reduction in the number of post views, these less popular languages offer a single solution in their standard libraries that doesn't depend nearly as much on either the context or data structures involved.

Without getting into why this difference might exist (although see note 1 below), we have a situation today where a large population of programmers have to either carry this complexity in their heads or disrupt their flow to look it up, and then make a suitable decision, just to do such a simple thing?

This might help us to understand why programmers are now using AI coding assistants to reduce context switching and work more efficiently on the actual problem they're trying to solve.

These new tools are feasible because datasets of questions and answers have grown over time to become large enough for LLMs to work effectively. However, I would submit that this growth has come not only from the popularity of these languages, but also from their shortcomings.

Wouldn't it be better if these simple things were made easy, to have the solutions at hand, without reaching for AI?

To be clear, I've seen more complex examples where AI assisted coding has significant benefits, but it wouldn't be needed in these simple cases if we stopped to reconsider our choices.

The alternative is surely a future where more and more output from these coding assistants ends up feeding the models used by those same assistants to serve these types of programming questions, and repeat. What could possibly go wrong? All this just to solve some simple problems.

A related issue concerning our choices is how well any given language is deemed to work with AI coding assistants. These tools might appear to be less effective for some languages because the LLMs don't have as much data to work with. However, we shouldn't naively assume this arises solely from a lack of popularity. We can see that some languages offer ready solutions for simple problems, and as a consequence we might expect considerably fewer questions and answers to be posted online in the first place.

I think this is an opportunity to reflect on what we are doing, what we are using and where we are going.

I think it's possible for us as programmers to do what's asked of us, and even do it with less disruption and more efficiency, but we can also focus on using AI where it actually adds value by enhancing our capabilities, instead of using it to compensate for the baggage brought by popular languages.

Programming could still be fun, rather than frustrating. Although if it's to be done by a machine, who cares right? Whilst there's no doubt that AI will fundamentally change the nature of programming in the coming years, through our choices and influence I hope you can see that the outcome for us is most definitely not inevitable.

Some thoughts to consider next time you're trying to do a simple thing and wondering why it's anything but!

Credits.

Igor Garcia for your feedback and advice: thank you 🙏

Notes.

1. I offer two hypotheses for why the most popular languages don't provide coverage for our case:

a) Due to incidental complexity resulting from their age and the prioritisation of backwards compatibility (the “shortcomings” and “baggage” I refer to in the article). The less popular languages I selected are all much younger so they can learn from the past and adopt more recent academic research, in particular concerning data structures, their abstractions and the functions that operate on them.

b) In recent times there has been a greater need for these kinds of functions over data, because even if you don't need it in your implementation you're more likely to need it in your tests. When these languages were conceived, even example-based unit testing wasn't a thing, but now we have property-based tests where the ability to generate data is essential.

2. What I present here isn't particularly new, except perhaps for the analysis and AI perspective. Essentially it's just another manifestation of what happens when programmer convenience and replaceability are prioritised above other concerns, as described by Rich Hickey in his 2011 presentation Simple Made Easy.

3. I'm not too familiar with Rust or Kotlin so if you think my analysis is incorrect, please let me know.

4. Yes, I did use an LLM-based generator for the image in the header but I'm fine with that, apart from the fact that it did not let me give any attribution or reward to the creators of the source images used by its model.

1. https://spectrum.ieee.org/top-programming-languages-2024↩
2. https://openjdk.org/jeps/461↩
3. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/groupBy↩
4. https://spectrum.ieee.org/top-programming-languages-2024↩
5. https://kotlinlang.org/api/core/kotlin-stdlib/kotlin.collections/chunked.html↩
6. https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.array_chunks↩
7. https://clojuredocs.org/clojure.core/partition-all↩