Book a Free Call
No commitment required
We’ve all been there: you ship new code, celebrate for a moment, then watch the error logs light up. Requests timing out, users complaining, someone getting paged at dinner. Sometimes it’s not your code. It’s just that the server quit before it could finish what it was doing.
In Go, implementing graceful shutdowns is remarkably straightforward, but the patterns aren’t always obvious to teams new to production Go services. This post explores practical techniques for building Go HTTP servers that shut down cleanly, completing active work before releasing resources. These patterns protect user experience during deployments, scaling events, and infrastructure maintenance.
When a Go server receives a termination signal, whether from a Kubernetes pod deletion, an EC2 instance replacement, or a manual restart, the default behavior is immediate process termination. This brutal shutdown ignores HTTP requests currently being processed, leaving clients with connection errors instead of responses.
Consider the business impact:
The solution lies in intercepting termination signals and orchestrating an orderly shutdown sequence. Go’s context package and http.Server provide the primitives; the pattern determines reliability.
This pattern has become table stakes for modern infrastructure. AWS, Google Cloud, and Azure all bake graceful shutdown expectations into their container orchestration platforms: ECS, GKE, and AKS assume your application handles SIGTERM within configurable windows. Frameworks like Kubernetes default to 30-second grace periods, while serverless platforms like AWS Fargate enforce stricter timeouts. The industry has converged: reliable shutdown handling is no longer exceptional engineering, it’s baseline operational hygiene.
Beyond the technical mechanics, graceful shutdowns reshape how teams work. When deployments stop being scary, release frequency increases. Developers push smaller changes more often, reducing the risk surface of each deployment. On-call engineers sleep better knowing a rolling restart won’t page them at 2 AM. Product managers stop scheduling launches around “safe deployment windows.” The organizational impact ripples outward: confidence compounds into velocity, and velocity into competitive advantage.
Graceful shutdowns begin with signal awareness. Go’s os/signal package allows applications to register interest in operating system signals: specifically SIGINT (Ctrl+C) and SIGTERM (the standard termination signal used by container orchestrators).
Here’s the foundational pattern:
package main
import (
"context"
"fmt"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
// Create a base context for the application
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT,
syscall.SIGTERM,
)
defer stop()
// Initialize your HTTP server
srv := &http.Server{
Addr: ":8080",
Handler: buildRouter(),
}
// Start server in a goroutine so shutdown can run concurrently
go func() {
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
fmt.Fprintf(os.Stderr, "Server error: %v\n", err)
}
}()
fmt.Println("Server ready on :8080")
// Wait for termination signal
<-ctx.Done()
fmt.Println("\nShutdown signal received, commencing graceful shutdown...")
// Create a timeout context for shutdown operations
shutdownCtx, cancel := context.WithTimeout(
context.Background(),
30*time.Second,
)
defer cancel()
// Attempt graceful shutdown
if err := srv.Shutdown(shutdownCtx); err != nil {
fmt.Fprintf(os.Stderr, "Graceful shutdown failed: %v\n", err)
// Force close if graceful shutdown fails
if closeErr := srv.Close(); closeErr != nil {
fmt.Fprintf(os.Stderr, "Forced close failed: %v\n", closeErr)
}
}
fmt.Println("Server stopped cleanly")
}
func buildRouter() http.Handler {
mux := http.NewServeMux()
mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("healthy"))
})
return mux
}
This foundation demonstrates several critical concepts:
Real-world applications require more sophisticated handling. Long-running requests, background workers, and external resource connections all need coordinated shutdown. Here’s a production-hardened pattern:
package main
import (
"context"
"database/sql"
"fmt"
"net/http"
"os"
"os/signal"
"sync"
"syscall"
"time"
_ "github.com/mattn/go-sqlite3"
)
// ResourceManager coordinates lifecycle of shared resources
type ResourceManager struct {
db *sql.DB
workers *WorkerPool
shutdown chan struct{}
wg sync.WaitGroup
}
func NewResourceManager() (*ResourceManager, error) {
db, err := sql.Open("sqlite3", "app.db")
if err != nil {
return nil, fmt.Errorf("database open: %w", err)
}
// Configure connection pool for graceful handling
db.SetMaxOpenConns(25)
db.SetMaxIdleConns(5)
db.SetConnMaxLifetime(5 * time.Minute)
return &ResourceManager{
db: db,
shutdown: make(chan struct{}),
}, nil
}
func (rm *ResourceManager) HealthCheck() error {
return rm.db.Ping()
}
func (rm *ResourceManager) Close() error {
close(rm.shutdown)
rm.wg.Wait() // Wait for background workers
return rm.db.Close()
}
type WorkerPool struct {
jobs chan func(context.Context)
}
func (wp *WorkerPool) Submit(job func(context.Context)) {
select {
case wp.jobs <- job:
default:
// Handle backpressure, log and potentially scale
fmt.Println("Worker pool saturated, consider scaling")
}
}
type Server struct {
httpServer *http.Server
resources *ResourceManager
}
func NewServer() (*Server, error) {
rm, err := NewResourceManager()
if err != nil {
return nil, err
}
s := &Server{
resources: rm,
}
router := http.NewServeMux()
router.HandleFunc("/health", s.healthHandler)
router.HandleFunc("/api/data", s.dataHandler)
router.HandleFunc("/api/process", s.processHandler)
s.httpServer = &http.Server{
Addr: ":8080",
Handler: router,
ReadTimeout: 5 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 120 * time.Second,
}
return s, nil
}
func (s *Server) healthHandler(w http.ResponseWriter, r *http.Request) {
if err := s.resources.HealthCheck(); err != nil {
http.Error(w, "unhealthy", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
w.Write([]byte("healthy"))
}
func (s *Server) dataHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// Simulate database work
row := s.resources.db.QueryRowContext(ctx, "SELECT datetime('now')")
var now string
if err := row.Scan(&now); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Write([]byte(fmt.Sprintf("Current time: %s", now)))
}
func (s *Server) processHandler(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// Check if shutdown has been initiated
select {
case <-ctx.Done():
http.Error(w, "server shutting down", http.StatusServiceUnavailable)
return
default:
}
// Long-running processing with context awareness
resultChan := make(chan string, 1)
go func() {
resultChan <- s.expensiveOperation(ctx)
}()
select {
case result := <-resultChan:
w.Write([]byte(result))
case <-ctx.Done():
// Client disconnected or shutdown initiated
http.Error(w, "request canceled", http.StatusGatewayTimeout)
}
}
func (s *Server) expensiveOperation(ctx context.Context) string {
select {
case <-time.After(3 * time.Second):
return "Processing complete"
case <-ctx.Done():
return "Operation canceled"
}
}
func (s *Server) Run() error {
return s.httpServer.ListenAndServe()
}
func (s *Server) Shutdown(ctx context.Context) error {
// Phase 1: Stop accepting new connections
fmt.Println("Shutting down HTTP server...")
if err := s.httpServer.Shutdown(ctx); err != nil {
return fmt.Errorf("http shutdown: %w", err)
}
// Phase 2: Close resources (database, workers, etc.)
fmt.Println("Closing resources...")
if err := s.resources.Close(); err != nil {
return fmt.Errorf("resource close: %w", err)
}
return nil
}
func main() {
// Setup signal handling
ctx, stop := signal.NotifyContext(context.Background(),
syscall.SIGINT,
syscall.SIGTERM,
)
defer stop()
// Initialize server with all resources
server, err := NewServer()
if err != nil {
fmt.Fprintf(os.Stderr, "Failed to initialize: %v\n", err)
os.Exit(1)
}
// Start server in background
go func() {
if err := server.Run(); err != nil && err != http.ErrServerClosed {
fmt.Fprintf(os.Stderr, "Server error: %v\n", err)
stop() // Trigger shutdown on unexpected error
}
}()
fmt.Println("Server ready on :8080")
// Wait for termination signal
<-ctx.Done()
fmt.Println("\nShutdown initiated...")
// Create shutdown context with timeout
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// Execute coordinated shutdown
if err := server.Shutdown(shutdownCtx); err != nil {
fmt.Fprintf(os.Stderr, "Shutdown error: %v\n", err)
os.Exit(1)
}
fmt.Println("Server stopped gracefully")
}
This production pattern introduces several sophisticated techniques:
The ResourceManager centralizes database connections, worker pools, and other shared resources. Its Close() method implements a two-phase shutdown: first signaling workers through a channel, then waiting for completion with sync.WaitGroup.
Every long-running operation accepts a context.Context, enabling cancellation propagation from the top-level shutdown signal down through database queries, HTTP requests, and background processing.
The health endpoint verifies database connectivity. During shutdown, health checks should fail to signal load balancers to stop routing traffic before the shutdown begins. Critical for zero-downtime deployments.
For high-throughput services, you may want visibility into shutdown progress:
func (s *Server) Shutdown(ctx context.Context) error {
// Get active connection count (requires custom ConnState tracking)
active := s.activeConnections.Load()
fmt.Printf("Draining %d active connections...\n", active)
done := make(chan struct{})
go func() {
s.httpServer.Shutdown(ctx)
close(done)
}()
ticker := time.NewTicker(5 * time.Second)
defer ticker.Stop()
for {
select {
case <-done:
return nil
case <-ticker.C:
remaining := s.activeConnections.Load()
fmt.Printf("Waiting for %d connections...\n", remaining)
case <-ctx.Done():
return ctx.Err()
}
}
}
Implementing active connection tracking requires http.Server.ConnState:
s.httpServer = &http.Server{
ConnState: func(conn net.Conn, state http.ConnState) {
switch state {
case http.StateNew:
s.activeConnections.Add(1)
case http.StateClosed, http.StateHijacked:
s.activeConnections.Add(-1)
}
},
}
Verify your implementation with integration tests:
func TestGracefulShutdown(t *testing.T) {
srv := NewTestServer()
go srv.Run()
// Send request that takes 2 seconds
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, _ := http.NewRequestWithContext(ctx, "GET", "http://localhost:8080/slow", nil)
// Trigger shutdown while request is in-flight
go func() {
time.Sleep(500 * time.Millisecond)
shutdownCtx, _ := context.WithTimeout(context.Background(), 10*time.Second)
srv.Shutdown(shutdownCtx)
}()
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Request should complete: %v", err)
}
if resp.StatusCode != http.StatusOK {
t.Errorf("Expected status 200, got %d", resp.StatusCode)
}
}
Not setting shutdown timeout: Without a deadline, a hung connection prevents deployment progress indefinitely.
Ignoring context cancellation: Background goroutines that don’t check ctx.Done() continue running after shutdown should complete.
Closing resources in wrong order: Closing the database before HTTP server shutdown causes in-flight requests to fail when they attempt database access.
Missing preStop hooks in Kubernetes: Without the delay, new requests arrive while you’re shutting down.
Graceful shutdowns transform deployments from disruptive events into seamless transitions. Go’s standard library provides robust primitives: signal.NotifyContext, http.Server.Shutdown, and context propagation, enabling sophisticated lifecycle management with minimal complexity.
The investment pays dividends: reduced error rates during deployments, improved user experience, and the confidence to release frequently. For teams building production Go services, graceful shutdown handling isn’t optional infrastructure. It’s essential reliability engineering.
Start with the basic pattern, add resource lifecycle management as your application grows, and tune Kubernetes configurations for your specific load balancer behavior. The result is a system that respects both your users’ requests and your team’s deployment velocity.