Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust regular expression for stripping non-utf8 characters #5

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Adjust regular expression for stripping non-utf8 characters #5

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 16 additions & 3 deletions opencalais.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ public function getEntities($document) {

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $this->api_url);
curl_setopt($ch, CURLOPT_HTTPHEADER,
curl_setopt($ch, CURLOPT_HTTPHEADER,
array(
'X-AG-Access-Token: ' . $this->api_token,
'Content-Type: ' . $this->contentType,
Expand All @@ -59,8 +59,21 @@ public function getEntities($document) {
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, $document);

$response = curl_exec($ch);
$response = preg_replace('/[\x00-\x1F\x80-\xFF]/', '', $response);
$response = curl_exec($ch);
// Remove non-utf8 characters from string, see http://stackoverflow.com/a/1401716
$regex = <<<'END'
/
(
(?: [\x00-\x7F] # single-byte sequences 0xxxxxxx
| [\xC0-\xDF][\x80-\xBF] # double-byte sequences 110xxxxx 10xxxxxx
| [\xE0-\xEF][\x80-\xBF]{2} # triple-byte sequences 1110xxxx 10xxxxxx * 2
| [\xF0-\xF7][\x80-\xBF]{3} # quadruple-byte sequence 11110xxx 10xxxxxx * 3
){1,100} # ...one or more times
)
| . # anything else
/x
END;
$response = preg_replace($regex, '$1', $response);

$object = json_decode($response);
if (empty($object)) {
Expand Down