Barry of Clearskys.net has found a fix for the spellcheck error. It was a problem with the headers sent by the script. Commenting the encoding header did the trick

//header('Content-Encoding: UTF-8');

the header is in the file rpc.php which is in wp-includes/js/tinymce/plugins/spellcheck