Okay tested now to turn off/on wifi while app is running and close it afterwards to start it again. No change. Maybe you mean something different with "remove from memory" than I do. I mean I double click home button and swipe app to the top of the iPad screen.
Unfortunately this approach does not solve the issue that easy remote cannot find the controller.