โWe just launched a 16TB archive of every dataset that has been available on data.gov since November. This will be updated day by day as new datasets appear. It can be freely copied, and we're sharing the code behind it to help others make their own archives of data they depend on.โ Harvard Library Innovation Lab (via BlueSky)
hey, thanks for doing this! Iโm super interested to see what youโre collected. Iโll just need to increase my storage capacity by a factor of thirty.
@freya recommend getting in touch with the LIL folks! (I was briefly affiliated but no longer am after my fellowship ended) lil.law.harvard.edu/contact/
Sign up for our newsletter or shoot us an email if youโre interested in any of our projects. All of our code is open source. Send a pull request on GitHub if you want to collaborate.
The map() function is used to apply a given function to every item of an iterable, such as a list or tuple, and returns a map object (which is an iterator). Let's start with a simple example of using map() to convert a list of strings into a list of
@flyingsquirrel Yup. And there's a multiprocessing version which can be useful if you're doing a very large csv but you probably want imap() for that one.
Source code: Lib/multiprocessing/ Availability: not Android, not iOS, not WASI. This module is not supported on mobile platforms or WebAssembly platforms. Introduction: multiprocessing is a package...
If you're looking for a function that applies another function to each element of a list and aggregates the results, it'd typically be done with a list comprehension:
[your_function(item) for item in your_list]
But you can also use list(map(your_function, your_list)) if you want. Basically, it's a built-in function rather than a method of the list class.
There's also a whole discussion to be had about lists vs generators and why you often wouldn't even need to make a list in the first place, but I won't get into that unless you want to know more.
the logic being that `map()` take any iterable, jot only lists, but sets, dicts, and anything that complies with the iterable protocol. I'm not saying you have to _like_ it. See also `str.join()`.
@Marcos Dione Yeah, in hindsight, that makes sense. I'm usually a Haskell programmer, not a Python programmer, so I don't usually have to deal with the question of whether a function is on an object or not.
I understand. I get similar issues when I jump to other languages. The ones I had more impedance has been golang for certain decisions and rust because of the "harsh" syntax and verboseness.
A new study published today suggests that up to 90% of cat owners experience oofjfjjggigiiiiifohhhjjfjfjjjjj)))))))))))))) the moment they leave their keyboard unattended. โDuring a new doublโฆ
LB I do not understand at all how people are arguing this is GTS behaving badly and not scrapers like...
FediDB: how many users do you have GTS: I'm not telling you don't crawl my page FediDB: I'm going to do it anyway how many users do you have GTS: Fifty billion FediDB: okay thanks FediDB: hey wait you lied to me!!!
Like idk sounds to me like you should have walked away at "don't crawl my page" and if you get the wrong numbers after that that's on you
Honestly looking at FediDBs GitHub there's a lot of complaints about it's scraping behavior in general. Like it will keep repeatedly hitting domains that don't even have fedi servers anymore
Is your feature request related to a problem? Please describe. It's quite silly that fedidb is slamming my server asking for instance info on instances I shut down years ago. I get requests every h...
Hi! I'm not very techy so this might be a dumb question, but. On your popular fediverse accounts list here: https://fedidb.org/popular-fediverse-accounts Where's George Takei? He's at @georgetakei@...
Ultimately the real problem we aren't discussing though is that Dansup is accusing the GTS devs of doing this "for clout", when they are doing this for "leave me alone", which is the opposite of clout.
It's complete bullshit and he should be embarrassed to say it.
Guy steadily gaining more influence over the fediverse with his software and massively unmanageable instance sizes spends most of his time beefing with other fedi devs...I don't think this is actually GOOD for the fediverse.
"This isn't about my scraper ignoring robots.txt and pounding instances that don't even exist anymore with requests indefinitely, it's about them trying to get clout by randomizing numbers (that I shouldn't have been scraping) so big nobody would even believe them!"
i said this before on a standalone post but if theres one thing he could learn from his idol gargron is to shut the fuck up and actually work on his bullshit software (that doesn't even do its job properly lmao)
seriously say what you want about website boy but at least he was smart enough to back off and do his shit instead of acting like the next jesus christ
@chkdsk it's funny because about a month ago during a meltdown Eugen literally told him DIRECTLY and PRIVATELY "hey man quit posting right now you're not doing yourself any favors" and he proceeded to post that message and keep posting for a bit and making everyone mad before he finally listened
A ton of his recent posts, including this GTS one, are full of nothing but people trying to tell him "hey man please stop posting this it's such a bad idea"
@jeremiah @chkdsk I think the posts he deletes have a bigger effect than the posts he keeps, because if he left half these posts up for people to see he'd get way less attention
personally while an instagram-like service has its use (visual artists would like it a lot) even though just making an alt frontend for mastodon/other existing softwares would do the job as well...
loops was a fucking grave mistake. not only because that form is an evil product of modern internet designed specifically to addict you and worsen your attention span, but also it just Won't fucking Work. tiktok is a cash cow because it has an algorithm designed to keep you sucked in for hours
so its just a dumb fucking thing that makes no sense. and i think posting short vids on existing microblogging side of fedi software is good as is
@chkdsk @jeremiah anyone who cares about fedi would have cloned vine instead
Small file sizes, hard to monetize so no big financial incentives, and in 6 seconds you can tell one tightly thought out joke or show your dog and can't spread weird misinformation
i haven't used the app myself but vines were actually fun. tiktok is just straight up brainrot (and while its sometimes funny it literally fucks up your brain)
@chkdsk @jeremiah out of the tiktoks I've seen the only good and memorable one I can think of that couldn't have fit in a vine is the "it goes in the square hole" one
@chkdsk @jeremiah I have a burner tiktok account that I only made to keep an eye on people buying plots of land in the desert or swamp attempting to build communes with absolutely no skills in anything
@chkdsk @jeremiah in a way, my interest in people building ill advised communes and my interest in people building ill advised online communities are the exact same interest, just one is more likely to end in someone going to the hospital
@gildilinie oh I've been into commune people on and off since well before Cohost the problem is you have to find the ones that spend enough time online that they post enough about it that you get the details
Luckily these days more and more ill advised communes are being started by tiktok people
@gildilinie Cohost should have been a commune tbh they at least had the money for property that isn't ten hours away from a water source and maybe some infrastructure
@chkdsk When you formulate it like that, yeah. You're right about loops.
I care more about Instagram because I've got artist friends who rely on Instagram for studios sales and courting galleries, and I mostly just want institutions and artists to be off big tech.
I've got a Pixelfed account on the flagship instance that I'm migrating now (didn't realize Dan never worked on meaningful export functionality until I went to use it).
I do like having a photo focused account/app separated from the microblogging one.
@jeremiah @chkdsk I keep wondering why we need servers dedicated to image posts instead of like...front ends
Maybe there's something I'm missing but it feels like you could just have Mastodon or GTS or whatever and just have a front end that filters down to just image posts, or a front end that shows your posts in a portfolio view
@jeremiah @chkdsk Most people I know who are trying Pixelfed (and are familiar with fedi meta in general) are trying it *despite* Dansup's posts, not because of them.
Making the pictures actually fit on my screen rather than posting through daily bouts of main character syndrome would be nice I guess, but what do I know ๐คท
It makes me think that his goals may be aligned with the "silicon valley exit strategy": - start any project - make drama - pose as a disruptor, opposed by various groups - inflate own audience numbers - sell
I don't run a fedi server, but I do run a regular website and any crawler that disobeys my robots.txt gets severely mistreated by me, without shame. If you break the well-established convention that robots.txt represents -- I don't care why -- then you get what you get.
This feels very firmly like the FO side of FA, FO.
A lot of people's first reaction seems to be "your server should not react to crawling this way" when I feel like the real question should be "why is the crawler not respecting robots.txt?", which just seems wild to me. Literally blaming the victim.
@Lnklnx :sdf: @lori If I'm not mistaken, one of the original purposes of robots.txt was to warn crawlers about things which might cause problems. If they ignore it, that's on them.
It haunts me that he didn't even say that, he said "spread tea," this language is so appropriated he doesn't even know how to work it properly.
Altho, speaking of language, he calls this an "inconvenience" here, when instead it's "lack of interest in shared values including on technical basics so basic even I, who doesn't touch computers, knows is basic."
@tully @bright_helpings and this is exactly why I feel strongly about calling this shit out at this point because he keeps saying shit and deleting it and thinking he won't be held accountable for anything. Unless you pay attention his timeline is basically always just good looking PR speak posts, but we all know that's not the reality
Right? I'm watching this whole thing and don't understand how it's a controversy because GTS is doing all the right things regarding this, but then again, the Fedi is crawling with tech bros. Some hide better than others.
What I don't get is why the same ire isn't being directed at Misskey and its forks? Misskey (and Sharkey) accurately report the total number of registered users, but they do not accurately report the monthly active users. On FediDB, some Misskey instances show 0 MAUs (even big instances with a lot of users), while some report a frozen value based on what their MAUs was before Misskey switched to not reporting it.
@2something if i had to really speculate, it's because the misskey devs didn't call dansup out on appropriating AAVE about a month ago, which he absolutely was doing
Just to provide some perspective here, the endpoint in question exists specifically to be crawled by robots, nodeinfo is not a user-facing page and does not serve a purpose for federation. GTS could remove the endpoint if theyโre not interested in it being crawled or serve 0s. The proposed behaviour seems to be more punitive than reasonable.
@Gargron but the real problem isn't what the software is doing, it's the extremely public accusation that another dev was acting maliciously at him personally
@chkdsk @jeremiah @Kiloku I'd say most people trying pixelfed have no idea who dansup is, most people couldn't tell you who runs any website unless they're billionaires. I guarantee most Tumblr users can't name the guy running that because he's not in the news every other day like Elon or Zuck. Dansup is a non entity to them.
TIL that my local library has a recording studio that you can book. I don't really have an immediate personal use for this, but it's pretty cool that it exists.
I know this is very first-world-problem, but now that I have my favourite #FountainPen back in working order, I had to make a tough decision about which pen gets the boot from the limited pen slots in my messenger bag. I imagine it's a bit like choosing favourites amongst one's children.
@R.L. Dane :Debian: :OpenBSD: ๐ต I've actually gone down the rabbit hole of potentially making one. I do have a sewing machine available. I just need to learn how to actually use one.
More dead tree spam from the Conservative party addressed to my father (who has never lived here). This one was labelled "compliance request" with a red military stencil looking font that said "URGENT: CODE RED". This speaks to the way they percieve the intelligence of their supporters. I weep for the state of politics in this country.
Is there a way to tell #Emacs #org-mode to omit yhe TOC and headline numbers when exporting to a text or markdown file? I'm trying to implement a #JohnnyDecimal system, so I'm supplying my own numbers and the 00.00-index.org file essentially is the table of contents.
Edit: Because not all replies federate, here's the solution I ended up with:
Sorry I don't have an answer. Nonetheless as an Emacs user (albeit only rarely in org-mode) I'll be interested in the responses you get. Cheers, -Randy
I got a new phone because my old one was giving out on me. This one is a decided downgrade from the last one, but that's (mostly) fine.
Anyhow, the new phone doesn't support wireless charging. Because of this, I can no longer charge it on the night stand breside my bed. When I wake up in the morning, my impulse in my half-awake state is to just grab my phone and go, forgetting that it's now tethered to the wall. This was not going to end well for anyone, most especially not my charging cable/port.
Auf YouTube findest du die angesagtesten Videos und Tracks. Auรerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.
We've trained an AI to replace your largest business cost, the CEO! We've carefully measured the outcomes of using this AI, and it's far more successful than humans making the same decisions! We've also trained AIs to replace your customers, your stockholders, and you. Please report to the AI replacement center.
I run a pretty tight budget. Twice a year, I get a fun surprise from my domain registrar (because I have two domains with them). They both come with a "free business card generator" that I have never used, but whatever.
Anyhow, every time a domain comes up for renewal, it's expected, budgeted, and paid for. All is good.
Then, about a month later I open my email and see an unexpected additional "receipt" from them. This sends me into a brief panic until I realize it's a $0 receipt for the business card bullshit.
Not my circus, not my monkeys: good luck with all that My circus, my monkeys: I am running a successful monkey circus Not my circus, my monkeys: I am running a successful monkey-rental service My circus, not my monkeys: oh no, oh man
hey everyone, noelle (a friend here in phoenix) needs $400 by sunday to not be homeless for February. it REALLY doesnt leave a lot of time but if youve got anything it would be greatly appreciated
Its been over a month since I was let go and I still haven't found a lot of prospects for a new job. If you know anyone who is looking for an #iOS Developer for remote work I really could use the contact. Please boost.
DOOM: The Gallery Experience lets you take part in an art gallery opening with a wine in your hand all while wandering through the Doomโs E1M1 map full of classic masterpieces.
Did you think every possible variation of the classic Doom game was explored already? Think again, because The Gallery Experience lets you take part in an art gallery opening with a wine in your hand all while wandering through the Doom's E1M1 map fuโฆ
Aargh! I thought I'd worked out a system to help with my #ADHD, and it was working for a while. Unfortunately, it involved using the alarm on my watch, which my brain is now just starting to filter out as background noise.
You don't have to post yours openly. I think there are various forums for sharing in a more limited manner, i.e. direct with someone. You can find addresses and mail to them first.
Pretty sure even short hand written notes count, so you could leave some fancy notes for people in your life who you appreciate.
The argument that people are responsible for filtering their own feeds and so Content Warnings are unnecessary doesn't work when many of the ones arguing this also constantly make up new fucking nicknames for the people they're talking about, requiring the constant addition of new filters to block out unfunny snarky bullshit.
I don't (and won't in certain cases) CW everything ever but if there's something I know SOMEONE out there might want to filter I try to get specific keywords in my post they might filter
Like for instance I wasn't gonna CW photos of my gerbils, I'm sure someone out there absolutely doesn't want to see rodents though, so I tried to be sure the word gerbil was in there
To some degree it involves some level of mind reading but it's something
even moreso on a platform where 99% of the platform doesn't have an easy way to search the text of your post intentionally obfuscating words is silly. Like I don't recommend posting the words KiwiFarms on Bluesky but you don't need to find 500 new ways to put an asterisk in homestuck on here
I get both sides of the CW debates and I have my own personal lines on what I find sensible or not, and on this account versus my other one I also accept that I agreed to follow certain CW rules on this instance and that's part of being a community even if I may not feel the same way on my own where I make the rules
But imo the best argument for NOT having to CW certain things is making an attempt to make sure people's word filters will work. You may not always get it right since like I said it takes a certain level of guesswork, and sometimes you just forget in the moment. But if you keep coming up with new nicknames for the worst people on earth it becomes impossible to keep up.
@Demon Queen Lucretia 2025 Edition I end up collapsing (auto-CWing) all of the offending user's posts. I don't know if Mastodon has this functionality though. A block would also be effective.
Though even still, it's sometimes a game of whack-a-mole.
Reminder that intolerance of intolerance is not a paradox. Tolerance is not an absolute ethical stance, it is a social contract. If someone chooses to not participate in the social contract of tolerance, they do not get to benefit from it.
Who was the genius that decided that Cult Pens' search bar should disappear as you're typing in it on mobile? This person belongs in UX jail. #FountainPens
EVHaste
in reply to Molly White • • •Kevin
in reply to Molly White • • •Annietime
in reply to Molly White • • •Pelican Dock Music
in reply to Molly White • • •Astrius
in reply to Molly White • • •quixote
in reply to Molly White • • •Mr. Lance E Sloan (IRL) ๐ค
in reply to Molly White • • •Orangenoel
in reply to Molly White • • •@lavaeolus
das schon gesehen hat?
โฐ๐ฒRandy Walters๐ฒโฐ
in reply to Molly White • • •Beachbum
in reply to Molly White • • •ZAC
in reply to Molly White • • •Publishers Clearing House
in reply to Molly White • • •Freya (it/its)๐ญ๐น๐ ๐ฉ
in reply to Molly White • • •Freya (it/its)๐ญ๐น๐ ๐ฉ
in reply to Molly White • • •Molly White
in reply to Freya (it/its)๐ญ๐น๐ ๐ฉ • • •Contact | Library Innovation Lab
lil.law.harvard.eduBurr Deming
in reply to Molly White • • •Beachbum
in reply to Molly White • • •xs4me2
in reply to Molly White • • •Blaise Pabรณn - controlpl4n3
in reply to Molly White • • •KChernecky For Democracy
in reply to Molly White • • •