Thanks. I'm on Debian, and this worked with some edits. The Ubuntu-intended files seem outdated in that some things gave warnings about incompatibilities.
New Ubuntu versions don't need this fix anymore.
This is what I ended up putting in /etc/fonts/conf.d/69-language-selector-ja-jp.conf
Code:
<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<!-- Japanese (ja) -->
<match target="pattern">
<test qual="any" name="family">
<string>serif</string>
</test>
<edit name="family" mode="prepend" binding="strong">
<string>DejaVu Serif</string>
<string>IPA P明朝</string>
<string>IPA モナー P明朝</string>
<string>Takao P明朝</string>
<string>梅P明朝</string>
<string>さざなみ明朝</string>
<string>東風明朝</string>
</edit>
</match>
<match target="pattern">
<test qual="any" name="family">
<string>sans-serif</string>
</test>
<edit name="family" mode="prepend" binding="strong">
<string>DejaVu Sans</string>
<string>HG丸ゴシックM-PRO</string>
<string>IPA Pゴシック</string>
<string>IPA モナー Pゴシック</string>
<string>Takao Pゴシック</string>
<string>UmePlus P Gothic</string>
<string>梅Pゴシック</string>
<string>VL Pゴシック</string>
<string>さざなみゴシック</string>
<string>東風ゴシック</string>
</edit>
</match>
<match target="pattern">
<test qual="any" name="family">
<string>monospace</string>
</test>
<edit name="family" mode="prepend" binding="strong">
<string>DejaVu Sans Mono</string>
<string>IPAゴシック</string>
<string>IPA モナー ゴシック</string>
<string>Takaoゴシック</string>
<string>UmePlus Gothic</string>
<string>梅ゴシック</string>
<string>VL ゴシック</string>
<string>さざなみゴシック</string>
<string>東風ゴシック</string>
</edit>
</match>
<match target="pattern">
<test qual="any" name="family">
<string>Ryumin</string>
</test>
<edit name="family" mode="prepend" binding="strong">
<string>IPA P明朝</string>
<string>IPA モナー P明朝</string>
<string>Takao P明朝</string>
<string>梅P明朝</string>
<string>さざなみ明朝</string>
<string>東風明朝</string>
</edit>
</match>
<match target="pattern">
<test qual="any" name="family">
<string>GothicBBB</string>
</test>
<edit name="family" mode="prepend" binding="strong">
<string>HG丸ゴシックM-PRO</string>
<string>IPA Pゴシック</string>
<string>IPA モナー Pゴシック</string>
<string>Takao Pゴシック</string>
<string>UmePlus P Gothic</string>
<string>梅Pゴシック</string>
<string>VL Pゴシック</string>
<string>さざなみ ゴシック</string>
<string>東風ゴシック</string>
</edit>
</match>
<!-- Japanese (ja) ends -->
Notice that I have added two custom fonts for serif and sans-serif. It's the HG fonts.