Go Channels slow down with more CPUs - Grant Stephens, Fastly

Go channels experience significant slowdown as CPU count increases - with CPU counts over 60, channel throughput becomes limited to around 1 million messages per second

The slowdown is caused by contention and lock overhead - when sending messages on channels, CPUs need to acquire locks, leading to competition between cores

Using buffered channels can help improve throughput at lower CPU counts but doesn’t solve the core scaling problem - performance still degrades with more CPUs

Solutions/workarounds explored:

• Setting GOMAXPROCS to limit CPU utilization
• Using runtime.LockOSThread() to pin goroutines to specific threads
• Implementing message buffering before channel sends
• Using multiple channels instead of a single bottleneck
• Adding timeouts for channel operations

For high throughput scenarios, batching multiple messages together before sending on channels can help reduce lock contention

Select statements have roughly half the performance of range loops when reading from channels due to checking multiple cases

Channel performance varies significantly between local development and production hardware - benchmarking on production hardware is important

Real-world workloads may need to consider architectural changes like sharding across multiple channels or implementing buffers rather than relying on raw channel performance

No silver bullet solution exists - different approaches work better for different use cases and tradeoffs need to be considered

New metrics added in Go 1.23 provide better visibility into channel wait times and lock contention