If you don't have a good use for them, it might be worth it to just sell sextants to other players, as they are quite valuable.
Other kinds of currency are used on items, but Sextants are used by applying them to maps on the Atlas. You unlock the Atlas after completing the 10 acts of the game's story.
Journeyman Cartographer's Sextants can be applied to both white and yellow maps, but it is more efficient to only use them on yellow ones, as white Sextants are cheaper and can also be applied to white maps.
Applying a Sextant to the Atlas adds an extra mod to the next three maps within its range that you open.
At first you can only apply one Sextant at a time. This limit can be raised by completing a whole tier (white, yellow, red) of maps or completing Zana's quests. Applying a sextant above the limit will remove the oldest one.
A common use for sextants is to roll for mods that add more monsters to the map, thus providing more loot.