Recall that the primary processing elements of the machine are the N64 CPU and the Reality CoProcessor (RCP). The CPU executes application code directly from the DRAM, transparently caching instruction and data references in on-chip caches. The code itself makes references to CPU virtual addresses, which are translated by on-chip hardware to physical memory addresses.
The RCP is primarily composed of two elements: the Signal Processor (SP) and the Display Processor (DP). The SP is a microcode engine that processes task lists for audio and graphics. The DP is, for the most part, driven by the SP. The RCP can be treated as a single processor for the purposes of memory management.
Finally, a number of DMA engines also access DRAM directly: the DP, as well as the Audio Interface (AI), Serial Interface (SI), and Parallel Interface (PI).
At the hardware level, all of these agents make references to physical DRAM addresses. These physical addresses are derived in very different ways, however.