Wow, lots of reflections you've inspired here. First of all, this highlighted part — this is sad, and I've seen it in ESL speakers a lot. I'm a former language teacher who has lived on 4 different continents, on top of having studied 3–5 other languages (depending on whether we're counting self-study or not) and multiple long-term relationships with ESL speakers, so I've had tons of experience with listening to (indeed, living within) other accents. Even still, there have been times when I meet people whose accent in English is SO hard to understand — for me: a very patient, multilingual, globally-aware person. And yet, when someone else can't understand them, they often default to, "I'm being discriminated against because I have an accent! They're pretending not to understand me!" Which, yes, there are assholes in this world who do that (It's been done to me too.)... but when even the most highly educated, multicultural, multilingual English speakers can't understand a given person, it's that much harder for the less cultured, and the non-native English speakers, to understand that same person. I feel especially bad for these ESL speakers when they are trying to go about their daily lives somewhere where English isn't the native language in the first place, because then they're that much less likely to be understood. (E.g., Berlin, where I live now.)
What I've come to realize is that being able to interpret foreign accents is often, counterintuitively, a mark of privilege. The people who struggle with this are usually people with lower educations or from insular, economically limited backgrounds who haven't had a lot of contact with foreigners, and therefore have had hardly any real-life practice in deciphering their speech patterns. By contrast, people with the privilege of a university education, the economic possibility of having lived in a major city, etc. — these people have lots of exposure to a diversity of accents and manage much better.
As long as you can be understood in your day-to-day life, then there's no need to knock yourself out devoting time to training your accent. Being multilingual, especially fluently so, is already a big achievement. Move on to the Italian, and don't worry about the English any further. :)