MIT Webscraper :)

3 pointsposted 12 hours ago
by ForgottenMail
(liamis.dev)

5 Comments

ventana

12 hours ago

  I was worried that getting this data would be difficult,
  but unironically, they literally have a single div for all
  of the text in the article and a single div for the header
  that contains the article name, author and date released.

  IT'S LIKE THEY WANTED ME TO SCRAPE THE SITE.
Why such a conclusion, and why is this "unironically"? A normal blog page, MIT or not, which is not obsessed with monetization or showing ads after each paragraph, is literally just a text with some very minor formatting (adding a few HTML tags here and there). It's very much expected that you would have a single <div class="content"> and an actual content inside of it.

ForgottenMail

11 hours ago

¯\(ツ)/¯ I was expecting it to be vauge and or have many divisions that would have made scraping harder, I was surprised that it did not. I also just have not had that much experience with large professional sites like that before

user

12 hours ago

[deleted]