This appears to be a bug.
I made my way into the mansion and tried to poison Yabu's tea but he never leaves the island. He's just standing there at the table looking around. The maid serving the tea is gone and it looks like there was no tea served at all. The maid in the courtyard requests the tea to be served from time to time but nothing happens.
I tried if I could shoot Yabu with Takuma but he does not react to the dog barking probably because he's too far away or he's just stuck in some loop.

Google didn't turn up anything so I started this thread. I will try to replay the mission from the start and see if it happens again.

Have you tried throwing rock with Hayato, or something else similiar? I had something like that happen, but they resumed to their usual routes after something interrupted their current thing.

It is a bug.

What I did is - run in fron of the Tea Caller (the female civilian in the Courtyard, that rings the first gong), then hide. Once the commotion quiets down - Yabu's routine should have reset to normal. (So basically any investigation/alarm you provoke may have the same result, doesn't have to be the Tea Caller - she just happened to be the only living thing left outside of Tea Island for me)

After a fresh start I could finish the mission normally.

So I tried a few things with my broken savegame.
Distracting the tea party with the whistle didn't phase Yabu much. He looked around and then back to his tea table.
Causing the maid in the court yard to call for help broke his loop once. But when he returned to his normal routine he and his guards got stuck again at the next waiting point (either checking for his barking dog or meeting up with his wife in the court yard). Then he would not even react to the maid calling for help just saying "I won't be disturbed". He would only break out of the loop when he himself saw me. Still his guards facing the other way didn't move an inch.