SquishyPillow@burggit.moe to Well That Was Useful!@burggit.moe · edit-21 year agoReddit scraper that works even after the API changescodeberg.orgexternal-linkmessage-square11fedilinkarrow-up19arrow-down10file-textcross-posted to: [email protected]
arrow-up19arrow-down1external-linkReddit scraper that works even after the API changescodeberg.orgSquishyPillow@burggit.moe to Well That Was Useful!@burggit.moe · edit-21 year agomessage-square11fedilinkfile-textcross-posted to: [email protected]
i started working on this after Reddit’s announcement about their API being paywalled. It still works lol
minus-squareBurger@burggit.moelinkfedilinkarrow-up1·1 year agoI’m sure they have automated requests detection and will put your IP on a cooldown, API or no API.
minus-squareMomoNeedsCorrection@burggit.moelinkfedilinkarrow-up3·1 year agoI mean given how much janky reddit seems to run at any given moment, I wouldn’t be surprised if it just didn’t work
minus-squareSquishyPillow@burggit.moeOPlinkfedilinkarrow-up2·1 year agoYou would be surprised. As long as you are using an approved User-Agent and are handling cookies correctly, it is pretty easy to scrape endlessly.
I’m sure they have automated requests detection and will put your IP on a cooldown, API or no API.
I mean given how much janky reddit seems to run at any given moment, I wouldn’t be surprised if it just didn’t work
You would be surprised. As long as you are using an approved User-Agent and are handling cookies correctly, it is pretty easy to scrape endlessly.