Update
I’ve answered my own question for the time being…
Following this answer + comment, you can wrap your algorithm environment like so:
\iffalse
\begin{verbatim}
\fi
\begin{algorithm}
...
\end{algorithm}
\iffalse
\end{verbatim}
\fi
which gets converted to
``` latex
\fi
\begin{algorithm}
...
\end{algorithm}
\iffalse
```
in my_file.md.
This method will preserve the LaTeX file for compilation to pdf, but the \iffalse statements are ignored by pandoc. It’s sketchy since it seems to rely on a bug, but it works. All that is left is to clean up the \fi and \iffalse lines leftover.
It’d be nice to have a more robust way of telling pandoc how to handle unknown environments though, or at least have it wrap them in a Div so you can process them with a filter.