Try to clear CSS cache, or disable it completely.

Also please specify full path to rtl.css file ?

To override system CSS you can add more complex rule, so it will have more priority, for example if system rule is looking like this:

.system-css {
    border:1px solid red;
}

then your rule with more priority should look like this:

html .system-css {
    border:1px solid green;
}