Try pulling the handbrake lever all the way up and then moving the whole thing forward.

 

Already did that... The photos above are with the highest I can pull the lever...

 

I have finally managed to do this, with a very risky trial-and-error.

There is a handbrake tension adjusting bolt and nut, right besides the handbrake lever. It is visible as soon as you pull out the cover over the lever.

I didn't know exactly what this part did, but I unscrewed the nut a lot. This made the lever go up higher than it was before, and i managed to get the rest of the console out.
I later found out that this bolt/nut regulates the amount of tension in the handbrake lever, so it was fine to unscrew it.

 

This was my next suggestion. It is important that after assembly it is tightened as it was before.

 

Thank you, zootv!
Yes. definitely needs to be tightened back after installation.
Another thing I realized, if anyone will attempt this: make sure your car wheels are blocked from rolling both backwards or forwards. A backup for the breaks. Some bricks will suffice but if you have any proper supports, do use them!

When you release the tension of the handbrake lever, it may also release the actual brakes.
I advise against just putting your car in gear as it can put stress over the mechanical assembly of the gears.