[SOLVED] Python .encode("utf-8") is causing lost of end of lines while writing to file

slimcharles · 01-03-2017, 10:20 AM

While scraping data with beautifulsoup,
The output is:

Code:

письмо 1
Test 2
Note 3
Über 4

But when I tried to write the output into file

Code:

with open("/home/user/Desktop/file", "a") as myfile:
    myfile.write(i['title'].encode("utf-8"))

the file's content is:

Code:

Yazı 1Test 2Note 3Über 4

As you can see the end of lines are missing.
Can you tell me what did I do wrong or not doing it ?

hydrurga · 01-03-2017, 10:27 AM

What code did you use to produce the output in box 1?

slimcharles · 01-03-2017, 10:39 AM

This is the code:

Quote:

Originally Posted by hydrurga

What code did you use to produce the output in box 1?

Code:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from bs4 import BeautifulSoup
import urllib2
import re

import sys
var = raw_input("Please enter something: ")
html_page = urllib2.urlopen("http://www.url.com/"+var+"files/")

soup = BeautifulSoup(html_page, "lxml")
li = soup.select("ul > li > a")
for i in soup.find('div',{'class':'boxList'}).findAll('a'):
      print i['title']
      with open("/home/user/Desktop/"+var, "a") as myfile:
        myfile.write(i['title'].encode("utf-8"))

hydrurga · 01-03-2017, 10:46 AM

The print() function automatically adds a newline on to the end of its output, unless you use the following format (for example):

print("Hello ",end='')
print("World")

(those are two single quotes that I've added after end=)

So, the newlines in your case have been artificially added to the output, they are not generated by beautifulsoup in the array entries. You therefore have to add newlines on to the end of each call to the write() function to do similar.

slimcharles · 01-03-2017, 11:01 AM

Quote:

Originally Posted by hydrurga

The print() function automatically adds a newline on to the end of its output, unless you use the following format (for example):

print("Hello ",end='')
print("World")

(those are two single quotes that I've added after end=)

So, the newlines in your case have been artificially added to the output, they are not generated by beautifulsoup in the array entries. You therefore have to add newlines on to the end of each call to the write() function to do similar.

I solved my problem by adding \n to the write function

Code:

myfile.write(i['title'].encode("utf-8") + "\n")

Thank you for your time and help.

hydrurga · 01-03-2017, 11:26 AM

No problem, slimcharles. I enjoy working with beautifulsoup (and requests too). Have fun!