I tried a few out-dated methods using xmodmap and getting ~/.xinitrc to run on startup but even if .xinitrc is executed, the key is still not mapped.
At the end I stumbled upon this answer when I was about to give up, and it worked.
I needed to change /usr/share/X11/xkb/symbols/pc with as root to modify the behavior of Insert key to Delete. Saved the file, reboot and it worked like a charm.
- key <INS> { [ Insert ] };
+ key <INS> { [ Delete ] };
