ListView and variable height items causes scroll problems

Question

I have a ListView that I use to scroll a list of fields, a text and a photo. Both heights are dynamic: The textview can contain a very long paragraph or just a sentence, and the image is scaled to fit width, so the height is variable (you can see the photo). Some items even don't have an image. This makes the items very heterogeneous in height. There is one item that can be 2 screens long, while another just one line of text.

I have it working etc.. but when I scroll upwards, as recycled views are inserted at the top, the scroll position jumps approximately the size of the new child. Scroll down works perfect...

I understand more or less why, obviously the recycled views are totally different in size, and this makes some scrolling algorythm fail, wherever it may be.

I have read about overriding itemType() and itemTypeCount, but I don't know if it's related to my problem. In the tests I did it didn't (magically) worked.

Can this be fixed easily, any trick or magic requestLayout here and there?

There's a guy suggesting to write a View recycler to fix this: question listview scrolls quite junky when therre are different row heights

Would writing a View recycler be as terribly complicated as it sounds?

Sorry for the abstract question..