1 // Copyright 2022 The Go Authors. All rights reserved. 2 // Use of this source code is governed by a BSD-style 3 // license that can be found in the LICENSE file. 4 5 package runtime 6 7 import "unsafe" 8 9 // OS memory management abstraction layer 10 // 11 // Regions of the address space managed by the runtime may be in one of four 12 // states at any given time: 13 // 1) None - Unreserved and unmapped, the default state of any region. 14 // 2) Reserved - Owned by the runtime, but accessing it would cause a fault. 15 // Does not count against the process' memory footprint. 16 // 3) Prepared - Reserved, intended not to be backed by physical memory (though 17 // an OS may implement this lazily). Can transition efficiently to 18 // Ready. Accessing memory in such a region is undefined (may 19 // fault, may give back unexpected zeroes, etc.). 20 // 4) Ready - may be accessed safely. 21 // 22 // This set of states is more than is strictly necessary to support all the 23 // currently supported platforms. One could get by with just None, Reserved, and 24 // Ready. However, the Prepared state gives us flexibility for performance 25 // purposes. For example, on POSIX-y operating systems, Reserved is usually a 26 // private anonymous mmap'd region with PROT_NONE set, and to transition 27 // to Ready would require setting PROT_READ|PROT_WRITE. However the 28 // underspecification of Prepared lets us use just MADV_FREE to transition from 29 // Ready to Prepared. Thus with the Prepared state we can set the permission 30 // bits just once early on, we can efficiently tell the OS that it's free to 31 // take pages away from us when we don't strictly need them. 32 // 33 // This file defines a cross-OS interface for a common set of helpers 34 // that transition memory regions between these states. The helpers call into 35 // OS-specific implementations that handle errors, while the interface boundary 36 // implements cross-OS functionality, like updating runtime accounting. 37 38 // sysAlloc transitions an OS-chosen region of memory from None to Ready. 39 // More specifically, it obtains a large chunk of zeroed memory from the 40 // operating system, typically on the order of a hundred kilobytes 41 // or a megabyte. This memory is always immediately available for use. 42 // 43 // sysStat must be non-nil. 44 // 45 // Don't split the stack as this function may be invoked without a valid G, 46 // which prevents us from allocating more stack. 47 // 48 //go:nosplit 49 func sysAlloc(n uintptr, sysStat *sysMemStat) unsafe.Pointer { 50 sysStat.add(int64(n)) 51 gcController.mappedReady.Add(int64(n)) 52 return sysAllocOS(n) 53 } 54 55 // sysUnused transitions a memory region from Ready to Prepared. It notifies the 56 // operating system that the physical pages backing this memory region are no 57 // longer needed and can be reused for other purposes. The contents of a 58 // sysUnused memory region are considered forfeit and the region must not be 59 // accessed again until sysUsed is called. 60 func sysUnused(v unsafe.Pointer, n uintptr) { 61 gcController.mappedReady.Add(-int64(n)) 62 sysUnusedOS(v, n) 63 } 64 65 // sysUsed transitions a memory region from Prepared to Ready. It notifies the 66 // operating system that the memory region is needed and ensures that the region 67 // may be safely accessed. This is typically a no-op on systems that don't have 68 // an explicit commit step and hard over-commit limits, but is critical on 69 // Windows, for example. 70 // 71 // This operation is idempotent for memory already in the Prepared state, so 72 // it is safe to refer, with v and n, to a range of memory that includes both 73 // Prepared and Ready memory. However, the caller must provide the exact amount 74 // of Prepared memory for accounting purposes. 75 func sysUsed(v unsafe.Pointer, n, prepared uintptr) { 76 gcController.mappedReady.Add(int64(prepared)) 77 sysUsedOS(v, n) 78 } 79 80 // sysHugePage does not transition memory regions, but instead provides a 81 // hint to the OS that it would be more efficient to back this memory region 82 // with pages of a larger size transparently. 83 func sysHugePage(v unsafe.Pointer, n uintptr) { 84 sysHugePageOS(v, n) 85 } 86 87 // sysFree transitions a memory region from any state to None. Therefore, it 88 // returns memory unconditionally. It is used if an out-of-memory error has been 89 // detected midway through an allocation or to carve out an aligned section of 90 // the address space. It is okay if sysFree is a no-op only if sysReserve always 91 // returns a memory region aligned to the heap allocator's alignment 92 // restrictions. 93 // 94 // sysStat must be non-nil. 95 // 96 // Don't split the stack as this function may be invoked without a valid G, 97 // which prevents us from allocating more stack. 98 // 99 //go:nosplit 100 func sysFree(v unsafe.Pointer, n uintptr, sysStat *sysMemStat) { 101 sysStat.add(-int64(n)) 102 gcController.mappedReady.Add(-int64(n)) 103 sysFreeOS(v, n) 104 } 105 106 // sysFault transitions a memory region from Ready to Reserved. It 107 // marks a region such that it will always fault if accessed. Used only for 108 // debugging the runtime. 109 // 110 // TODO(mknyszek): Currently it's true that all uses of sysFault transition 111 // memory from Ready to Reserved, but this may not be true in the future 112 // since on every platform the operation is much more general than that. 113 // If a transition from Prepared is ever introduced, create a new function 114 // that elides the Ready state accounting. 115 func sysFault(v unsafe.Pointer, n uintptr) { 116 gcController.mappedReady.Add(-int64(n)) 117 sysFaultOS(v, n) 118 } 119 120 // sysReserve transitions a memory region from None to Reserved. It reserves 121 // address space in such a way that it would cause a fatal fault upon access 122 // (either via permissions or not committing the memory). Such a reservation is 123 // thus never backed by physical memory. 124 // 125 // If the pointer passed to it is non-nil, the caller wants the 126 // reservation there, but sysReserve can still choose another 127 // location if that one is unavailable. 128 // 129 // NOTE: sysReserve returns OS-aligned memory, but the heap allocator 130 // may use larger alignment, so the caller must be careful to realign the 131 // memory obtained by sysReserve. 132 func sysReserve(v unsafe.Pointer, n uintptr) unsafe.Pointer { 133 return sysReserveOS(v, n) 134 } 135 136 // sysMap transitions a memory region from Reserved to Prepared. It ensures the 137 // memory region can be efficiently transitioned to Ready. 138 // 139 // sysStat must be non-nil. 140 func sysMap(v unsafe.Pointer, n uintptr, sysStat *sysMemStat) { 141 sysStat.add(int64(n)) 142 sysMapOS(v, n) 143 } 144