When participating in AMMs, you will always incur some impermanent loss. This occurs not only when prices go down, but also up: the variation of price between the two paired tokens causes IL. It is called impermanent, because it's a temporary state which may improve (or worsen) over time. The loss is not permanent until crystallized by exiting the pool.