Java for data engineers · JVM-02
Collections and streams
- 3 h
- requires JVM-01
From your world. a GROUP BY / window aggregation you'd write in SQL, expressed as a typed stream pipeline
1. "Walk me through a stream pipeline as a SQL query."
stream().filter(p)isWHERE p..map(f)isSELECT f(x)..flatMap(f)isUNNEST..collect(groupingBy(k, summingDouble(v)))isGROUP BY kwithSUM(v)..sorted(c)isORDER BY..distinct()isDISTINCT. Intermediate ops are lazy. The terminal op runs the pipeline.
2. "map vs flatMap?"
mapis 1→1 (project).flatMapmaps each element to a stream and flattens the streams, so it is 1→many. Examples: split a CSV line into tokens, or explode an array column.
3. "How do you count occurrences per key?"
Use
map.merge(key, 1, Integer::sum). It inserts 1 if the key is absent. Otherwise it adds 1. You can also usestream().collect(Collectors.groupingBy(k, Collectors.counting())). Avoidmap.put(k, map.get(k)+1). On the first occurrence,getreturnsnull, and the+1throws an NPE.
4. "When is a parallel stream a bad idea?"
It is a bad idea in three cases:
- The input is small (the pool overhead is greater than the benefit).
- The lambda has side effects or shared mutable state.
- The
reducecombiner is not associative.All three give wrong or non-deterministic results. Real parallelism belongs to Spark and Flink, not to a
parallelStream().
5. "Why doesn't my second use of a stream work?"
The terminal op consumes a stream. You cannot reuse it. Java reports "stream has already been operated upon or closed". Build a fresh stream from the source
List. Or collect to aListand reuse that.
6. "List.of(...) then .add(...) throws. Why, and the fix?"
List.ofreturns an immutable list. If you change it, Java throwsUnsupportedOperationException. When you need to add or remove elements, wrap it:new ArrayList<>(List.of(...)).