Skip to content
Snippets Groups Projects
Commit 2d0b6222 authored by James B. Pollack's avatar James B. Pollack Committed by Jeff Wu
Browse files

Update encoder.py to work on windows

This fixes https://github.com/openai/gpt-2/issues/26

```  File "C:\Users\James Pollack\Desktop\gpt-2\src\encoder.py", line 112, in get_encoder
    bpe_data = f.read()
  File "C:\Anaconda\envs\gpt-2\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 66951: character maps to <undefined>```
parent ebd5894d
No related branches found
No related tags found
No related merge requests found
......@@ -108,7 +108,7 @@ class Encoder:
def get_encoder(model_name):
with open(os.path.join('models', model_name, 'encoder.json'), 'r') as f:
encoder = json.load(f)
with open(os.path.join('models', model_name, 'vocab.bpe'), 'r') as f:
with open(os.path.join('models', model_name, 'vocab.bpe'), 'r', encoding="utf-8") as f:
bpe_data = f.read()
bpe_merges = [tuple(merge_str.split()) for merge_str in bpe_data.split('\n')[1:-1]]
return Encoder(
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment