All the functions for opening the suitelet page are the same as the normal procedure.
In the client script instead of using window.open use the below snippet.
function testFunction(obj) {
return (window.newPopUp = nlExtOpenWindow(obj.pathUrl, obj.name, obj.width, obj.height, {}));
}