With retaining walls, one big consideration is hydrostatic pressure. If water is allowed to collect behind a wall, the weight of saturated soil could topple it.
What goes behind a retaining wall matters just as much as the blocks you can see, and choosing the wrong material could undermine drainage and stability.