File I/O in Go

Reading Files

Read entire file into memory

Reading a whole file at once is the simplest approach and the right choice for configuration files, templates, and any file small enough to fit comfortably in memory. os.ReadFile (available since Go 1.16, replacing the deprecated ioutil.ReadFile) returns the file’s contents as a byte slice in a single call. If the file does not exist or cannot be read, it returns a descriptive error you can wrap with context.

import "os"

// os.ReadFile reads the entire file into a []byte in one call
data, err := os.ReadFile("data.txt")
if err != nil {
    return fmt.Errorf("reading file: %w", err)
}
fmt.Println(string(data))

Read line by line with bufio.Scanner

For large files — log files, CSV exports, multi-gigabyte datasets — loading everything into memory at once is impractical. bufio.Scanner reads one line at a time, keeping only a single line in memory regardless of the file’s total size. It also handles both Unix (\n) and Windows (\r\n) line endings automatically.

import (
    "bufio"
    "os"
)

func readLines(path string) ([]string, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("open %q: %w", path, err)
    }
    defer f.Close() // always close the file when done

    var lines []string
    scanner := bufio.NewScanner(f)
    for scanner.Scan() {
        lines = append(lines, scanner.Text()) // Text() returns the line without the newline
    }
    return lines, scanner.Err() // scanner.Err() is nil on EOF — check it for real errors
}

For files with very long lines, increase the scanner buffer to avoid “token too long” errors:

scanner := bufio.NewScanner(f)
scanner.Buffer(make([]byte, 1024*1024), 1024*1024) // allow lines up to 1 MB

Read in chunks with bufio.Reader

When you need finer control than line-by-line — for example, reading binary files or implementing a custom protocol parser — bufio.Reader lets you read up to a delimiter, read a fixed number of bytes, or read character by character, all with an internal buffer that minimises system calls.

reader := bufio.NewReaderSize(f, 64*1024) // 64 KB internal buffer

for {
    line, err := reader.ReadString('\n') // read up to (and including) the next newline
    if len(line) > 0 {
        process(line) // process even a partial line before checking err
    }
    if err == io.EOF {
        break // normal end of file
    }
    if err != nil {
        return err // real read error
    }
}

Writing Files

Write entire content at once

os.WriteFile is the counterpart to os.ReadFile: it writes a byte slice to a file in one call, creating the file if it doesn’t exist or truncating it if it does. The permission bits (e.g. 0644) apply when the file is newly created. For append-only patterns like log files, use os.OpenFile with the O_APPEND flag.

// Write entire content atomically — creates or truncates the file
content := []byte("Hello, Go!\nSecond line\n")
err := os.WriteFile("output.txt", content, 0644) // 0644 = owner rw, group r, other r
if err != nil {
    return err
}

// Append to an existing file — useful for log files
f, err := os.OpenFile("log.txt", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
    return err
}
defer f.Close()
_, err = f.WriteString("new log entry\n")

Buffered writes for many small writes

Writing many small strings one at a time with f.Write generates a system call for each write — this is slow. bufio.Writer batches those writes into its internal buffer and flushes to disk in large chunks. The critical detail: you must call Flush() when done, otherwise data sitting in the buffer never reaches disk.

f, err := os.Create("output.txt")
if err != nil {
    return err
}
defer f.Close()

w := bufio.NewWriter(f)
for i := 0; i < 1000; i++ {
    fmt.Fprintf(w, "line %d\n", i) // writes to the buffer, not to disk yet
}
w.Flush() // flush the buffer to disk — REQUIRED, or data is lost

Working with Paths

Cross-platform path handling is one of the quiet sources of bugs in Go programs. Hard-coding forward slashes breaks on Windows; filepath.Join builds the correct separator for the current OS automatically. Always use filepath functions for filesystem paths, and path (no filepath) only for URL-style paths that are always forward-slash separated.

import (
    "os"
    "path/filepath"
)

// filepath.Join builds the correct separator for the OS
path := filepath.Join("data", "users", "alice.json")
// Windows: data\users\alice.json
// Linux/Mac: data/users/alice.json

// Extract components from a path
dir  := filepath.Dir("/home/alice/file.txt")  // /home/alice
base := filepath.Base("/home/alice/file.txt") // file.txt
ext  := filepath.Ext("/home/alice/file.txt")  // .txt

// Check if a file exists — use os.Stat, not a separate Exists function
info, err := os.Stat("myfile.txt")
if os.IsNotExist(err) {
    fmt.Println("file does not exist")
} else if err == nil {
    fmt.Printf("size: %d bytes, modified: %v\n", info.Size(), info.ModTime())
}

// Walk a directory tree recursively
filepath.WalkDir(".", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err // stop walking on permission errors
    }
    if !d.IsDir() && filepath.Ext(path) == ".go" {
        fmt.Println(path)
    }
    return nil
})

// Create a full directory path (like mkdir -p)
os.MkdirAll("data/cache/temp", 0755)

// Create a temporary file — OS cleans it up eventually, but delete it yourself
f, err := os.CreateTemp("", "prefix-*.tmp")
defer os.Remove(f.Name()) // remove when done

JSON Encoding and Decoding

Encode (Marshal)

Go’s encoding/json package uses struct field tags to control JSON key names, omit empty fields, and exclude sensitive data. Struct tags are the idiomatic way to bridge Go’s camelCase/PascalCase naming conventions with JSON’s snake_case conventions. The json:"-" tag completely excludes a field — use it for passwords, tokens, and other data that must never appear in API responses.

import "encoding/json"

type User struct {
    ID        int       `json:"id"`
    Name      string    `json:"name"`
    Email     string    `json:"email,omitempty"` // omitted if empty string
    CreatedAt time.Time `json:"created_at"`
    Password  string    `json:"-"` // never serialised — excluded from all JSON output
}

user := User{ID: 1, Name: "Alice", Email: "[email protected]"}

// Compact JSON — smallest size, for network transmission
data, err := json.Marshal(user)
fmt.Println(string(data))
// {"id":1,"name":"Alice","email":"[email protected]","created_at":"0001-01-01T00:00:00Z"}

// Pretty-printed JSON — indented for human readability or config files
pretty, err := json.MarshalIndent(user, "", "  ")
fmt.Println(string(pretty))
// {
//   "id": 1,
//   "name": "Alice",
//   ...
// }

Decode (Unmarshal)

json.Unmarshal parses a JSON byte slice into a Go value. It matches JSON keys to struct fields case-insensitively and ignores unknown keys by default — making it resilient to API changes. Always check the returned error: malformed JSON, wrong types, and overflowed numbers all surface here.

jsonStr := `{"id": 1, "name": "Alice", "email": "[email protected]"}`

var user User
err := json.Unmarshal([]byte(jsonStr), &user) // pass a pointer so Unmarshal can fill the struct
if err != nil {
    return fmt.Errorf("parsing user: %w", err)
}
fmt.Println(user.Name) // Alice

// Decode into map when the structure is unknown or dynamic
var raw map[string]any
json.Unmarshal([]byte(jsonStr), &raw)
fmt.Println(raw["name"]) // Alice

Streaming JSON (large files)

When working with large JSON datasets, loading everything into memory before encoding or decoding is wasteful. json.NewEncoder and json.NewDecoder operate on io.Writer and io.Reader interfaces directly, so they can process data item by item. This is the right approach for writing JSONL (one JSON object per line) log files or reading API responses that return arrays of thousands of objects.

// Write many records one at a time — no need to build a giant slice first
f, _ := os.Create("records.jsonl")
defer f.Close()

enc := json.NewEncoder(f)
for _, record := range records {
    enc.Encode(record) // writes one JSON object followed by a newline
}

// Read many records one at a time — only one record in memory at once
f, _ = os.Open("records.jsonl")
defer f.Close()

dec := json.NewDecoder(f)
for dec.More() { // More() returns false when there are no more values to decode
    var record Record
    if err := dec.Decode(&record); err != nil {
        return err
    }
    process(record)
}

Practical Example — Config File Read/Write

This example combines file I/O and JSON into a complete config management pattern. Notice the file permission 0600 on write — config files often contain secrets, so owner-only read/write access is appropriate. Wrapping errors with fmt.Errorf and %w at each step makes it clear whether the failure was in reading, parsing, or encoding.

package main

import (
    "encoding/json"
    "fmt"
    "os"
)

type AppConfig struct {
    Database struct {
        Host     string `json:"host"`
        Port     int    `json:"port"`
        Name     string `json:"name"`
    } `json:"database"`
    Server struct {
        Port    int    `json:"port"`
        TLSCert string `json:"tls_cert,omitempty"`
    } `json:"server"`
    Debug bool `json:"debug"`
}

func loadConfig(path string) (*AppConfig, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("loading config: %w", err)
    }

    var cfg AppConfig
    if err := json.Unmarshal(data, &cfg); err != nil {
        return nil, fmt.Errorf("parsing config: %w", err)
    }
    return &cfg, nil
}

func saveConfig(path string, cfg *AppConfig) error {
    data, err := json.MarshalIndent(cfg, "", "  ") // pretty-print for human readability
    if err != nil {
        return fmt.Errorf("encoding config: %w", err)
    }
    return os.WriteFile(path, data, 0600) // 0600 = owner read/write only — config may contain secrets
}

func main() {
    cfg := &AppConfig{Debug: true}
    cfg.Database.Host = "localhost"
    cfg.Database.Port = 5432
    cfg.Database.Name = "myapp"
    cfg.Server.Port = 8080

    if err := saveConfig("config.json", cfg); err != nil {
        fmt.Println(err)
        return
    }

    loaded, err := loadConfig("config.json")
    if err != nil {
        fmt.Println(err)
        return
    }
    fmt.Printf("DB: %s:%d/%s\n", loaded.Database.Host, loaded.Database.Port, loaded.Database.Name)
    // DB: localhost:5432/myapp
}