Line Filters
Reading from os.Stdin with bufio.Scanner, writing to os.Stdout, processing lines with strings operations, and the Unix filter pattern.
A line filter reads lines from standard input, transforms them, and writes results to standard output. This is the Unix filter pattern - Go binaries that follow it compose naturally with grep, awk, sed, and other shell tools.
A minimal line filter: read every line from os.Stdin with bufio.Scanner and write it to os.Stdout unchanged. This is the identity filter - the template you extend to add transformations.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}Add a transformation to produce a useful filter. This example uppercases every line - a trivial but illustrative change. Replace the strings.ToUpper call with any line-level logic you need.
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
fmt.Println(strings.ToUpper(scanner.Text()))
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}Filters can also read from a file argument and fall back to os.Stdin when none is provided - the convention followed by most Unix tools. os.Stdin implements io.Reader, so bufio.NewScanner accepts it directly.
package main
import (
"bufio"
"fmt"
"io"
"os"
"strings"
)
func filter(r io.Reader) {
scanner := bufio.NewScanner(r)
for scanner.Scan() {
fmt.Println(strings.ToUpper(scanner.Text()))
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "error:", err)
os.Exit(1)
}
}
func main() {
if len(os.Args) > 1 {
f, err := os.Open(os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer f.Close()
filter(f)
} else {
filter(os.Stdin)
}
}In production
Line filters are the building blocks of shell pipelines - a Go binary that reads stdin and writes stdout composes with grep, awk, and jq without extra plumbing. If you wrap os.Stdout in a bufio.Writer for performance, call w.Flush() before os.Exit or returning from main - the OS flushes os.Stdout on normal process exit, but buffered writes in a bufio.Writer are not automatically flushed. Write errors to os.Stderr, not os.Stdout, so they do not corrupt the output stream in a pipeline. Exit with a non-zero code on error (os.Exit(1)) so downstream tools and shell scripts can detect failure. bufio.Scanner has a 64 KB default line limit; call scanner.Buffer before the first Scan if your input lines can exceed that.
Enjoyed this? Get more essays on software craft delivered to your inbox.
Subscribe free