27 minutes ago
Ive only used reduce at work half a dozen times and it does raise an eyebrow each time.
But for unioning a bunch of spark dataframes together i think
df = reduce(DataFrame.union, list_of_dfs)
df, *rest = list_of_dfs
for other in rest:
df = df.union(other)
7 hours ago
I think this is because in an imperative language, `reduce` does not actually give you much over a `for item in collection` loop. With `map` and `filter`, you immediately learn something about the result (it's a list of the same length as the original, with each item only depending on the corresponding original item; it's a list containing some of the original elements unchanged and nothing else). This is useful, so `map` and `filter` are good.
With `reduce`, the result could be anything, and in an imperative language, side effects are also possible. So it's just a loop with worse syntax.
(Admittedly, in an imperative language, `map` and `filter` could also have side effects, though I think most people would consider this bad style.)
3 hours ago
I think what makes reduce less popular is that it takes two lambdas:
- a slightly awkward one that takes a partial result and the next value to produce a new partial result
- one that maps the final partial result to the result
Also, in many languages, when reading the code, you have to skip initialization of the partial result, read the lambda, and then jump back to make sense of the initial values
I think something like awk’s syntax, with BEGIN and END blocks would improve on that. Example of a first go at such syntax (needs work):
Items.BEGIN
min = ∞
max = -∞
sum = 0
n = 0
ITER
min = Min(min,_)
max = Max(max,_)
n += 1
sum += _
RETURN
average = sum / n
(min, max, average)
- items in the partial results have names, making them easier to understand
- result also is easier to understand
Price paid is wordiness, and you cannot simply write a function name for either of the lambdas.
However, I think the latter only is useful in case the partial result is the final result. There, you can keep
sum = items.reduce(0,+)
6 hours ago
I think that in every imperative language that offers `map`, `filter`, `reduce`, or similar, the written contract of this API should state that any higher-order function handed to it as an argument must be free from side effects.
I think I’ve seen several language core APIs have this in their contract, e.g. `Stream#reduce` in Java [0] (emphasis mine):
> accumulator - an *associative, non-interfering, stateless* function for combining two values
[0]: https://docs.oracle.com/javase/8/docs/api/java/util/stream/S...
an hour ago
In Rust these specifically take `FnMut`, a function which can update internal/borrowed state, rather than `Fn` which can't easily. In `map` or `filter` you shouldn't rely on the iteration order so that's not often useful – maybe something 'logically' stateless but which needs a mutable connection/threadpool/cache, or eg a counter which is really an ancillary reduction. There's even `inspect` which is explicitly for such side effects. In `fold`, the order is guaranteed and you could use it for a state machine, a fiddly `zip` with other mutable iterators, etc – something you need to perform the reduction, but which isn't really an output, I think you could reasonably write either
.fold(init, move |acc, x| {…}) // or
.fold((state, init), |(state, acc), x| {…}).12 hours ago
I mean, most of the code that I write would be side-effect free anyway. In an imperative loop, this would also be true except for updating local variables. If this is the case, `reduce` really is the same as a loop over a collection, except that the names for the state passed between iterations come out better. In the `reduce` version, you can name the parameters to the reducer, but often not the return values. As a reader, one needs to connect the return values to the parameters by position.
(Note that by "loop over a collection", I explicitly mean a looping construct that gives the elements of the collection directly, instead of looping over indices and extracting the elements manually.)
4 hours ago
Even though it makes print debugging harder, I think it would be better if the language enforced such a contract.
2 hours ago
I was going to write a question asking if reduce is the thing I know as accumulate (I think I picked this up from SICP). But then I went to wikipedia, and it seems that an even more common name is fold.
Here's a hypothesis: The fact that the same operation has half a dozen different names makes it sound like there is a lot to learn. If I am totally familiar with fold, and i come upon a reduce, I may need to think more about what's going on, which is distracting.
I don't think map and filter have so many synonyms? I know select for filter, but it seems to me less common.
an hour ago
And in some contexts you have the subtle distinction that fold is linear and reduce requires an associative operation and an identity element (aka a monoid)
an hour ago
For can have another set of variables in the header too. You can simulate it more readably even if you need to call the lambda.