Not really, it can happen anywhere with a transition to another place/actions/people/in time. A cliffhanger doesn't need to happen at the end of a chapter.
Just as an exemple : at the start of a chapter Sara says she'll be absent from school for the next day. When she leaves we see a car rushing where she's at when she cross the road then the story shift to the MC at school. And we only know later on what really happend.
That is a narative cliffhanger, and it doesn't need to be at the end of a chapter, and there is many more ways to make cliffhangers in a story/chapter. Cliffhangers are it's true mostly used at the end of chapters and stories. But it's used too widely, too often and only in the same way. It really became boring in most VN as the only way they find to keep people hyped for the next chapter are cliffhangers. A series of hints/clues without follow up until later, can also be a cliffhanger.