BeautifulSoup возвращает TypeError: объект NoneType не имеет len()

Я использую BeautifulSoup для очистки данных и возврата мне списка всех div в моем списке, но это дает мне эту ошибку:

Traceback (most recent call last):
  File "C:\Users\intel\Desktop\One page\test.py", line 16, in <module>
    soup = BeautifulSoup(div.html,'html5lib')
  File "C:\Users\intel\AppData\Local\Programs\Python\Python38\lib\site-packages\bs4\__init__.py", line 287, in __init__
    elif len(markup) <= 256 and (
TypeError: object of type 'NoneType' has no len()

Вот мой код:

from bs4 import BeautifulSoup
import requests as req

resp = req.get('https://medium.com/@daranept27')

x = resp.text

soup = BeautifulSoup(x, "lxml")
 
divs = soup.find_all("div")
#print(divs)

lst = []

for div in divs:
    soup = BeautifulSoup(div.html,'html5lib')
    div_tag = soup.find()
    try:
        title = div_tag.section.div.h1.a['href']
        if title not in lst: lst.append(title)
    except:
        pass

print("\n".join(lst))

python-3.x python web-scraping python-requests beautifulsoup
person Community    schedule 06.11.2020    source источник

Ответы (1)


arrow_upward
0
arrow_downward

Попробуйте использовать str(div) для преобразования div в str. Вот полный код:

from bs4 import BeautifulSoup
import requests as req

resp = req.get('https://medium.com/@daranept27')

x = resp.text

soup = BeautifulSoup(x, "lxml")

divs = soup.find_all("div")
# print(divs)

lst = []

for div in divs:
    soup = BeautifulSoup(str(div), 'html5lib')
    div_tag = soup.find()
    try:
        title = div_tag.section.div.h1.a['href']
        if title not in lst: lst.append(title)
    except:
        pass

print("\n".join(lst))

Выход:

/read-rosy/if-the-whole-world-is-compelled-to-forget-everything-cde200c0ad98
/wordsmith-library/seven-days-between-life-and-death-dffb639fb245
/an-idea/have-you-ever-encountered-a-fake-friend-if-so-try-these-simple-tips-to-overcome-it-d8473d755ab8
person Sushil    schedule 06.11.2020